[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-09-05 Thread Bing Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154766#comment-16154766
 ] 

Bing Li commented on HIVE-10567:


[~ashutoshc] I found that the previous patch could work on the latest master 
branch. So I just added the test case without any other code changes.

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Bing Li
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch, HIVE-10567.2.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-07-20 Thread Bing Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094519#comment-16094519
 ] 

Bing Li commented on HIVE-10567:


Hi, [~ashutoshc]
The fix doesn't affect because the variable partitions is empty.

{code:java}
  case DYNAMIC_PARTITION:
for (Partition 
dynPart:tableScanOp.getConf().getTableMetadata().getTableSpec().partitions) {
  inputPaths.add(dynPart.getDataLocation());
}
break;
{code}

The "partitions" is set in Hive.getPartitionsByNames().

{code:java}
  public List getPartitionsByNames(Table tbl,
  Map partialPartSpec)
  throws HiveException {

if (!tbl.isPartitioned()) {
  throw new HiveException(ErrorMsg.TABLE_NOT_PARTITIONED, 
tbl.getTableName());
}

   // the size of names is 0;
List names = getPartitionNames(tbl.getDbName(), tbl.getTableName(),
partialPartSpec, (short)-1);

List partitions = getPartitionsByNames(tbl, names);
return partitions;
  }
{code}



 

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-07-19 Thread Ashutosh Chauhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093572#comment-16093572
 ] 

Ashutosh Chauhan commented on HIVE-10567:
-

yes.. i also tested the patch uploaded and found it doesn't fix the issue. Let 
me know what you find. It will be good to fix this.

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-07-19 Thread Bing Li (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093552#comment-16093552
 ] 

Bing Li commented on HIVE-10567:


[~ashutoshc] and [~tfriedr] I tried to apply the current patch to master and 
ran the following query:

analyze table analyze_srcpart_partial_scan partition(ds, hr) compute statistics 
partialscan;

It still reported the ERROR:

{code:java}
java.io.IOException: No input paths specified in job
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:472)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:502)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:329)
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:320)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
{code}

I will take a look at the reason later.


> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-07-07 Thread Ashutosh Chauhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078245#comment-16078245
 ] 

Ashutosh Chauhan commented on HIVE-10567:
-

[~tfriedr] Would you like to add a test case for this in stats_partscan_1_23.q 
qfile?

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-07-06 Thread Ashutosh Chauhan (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076672#comment-16076672
 ] 

Ashutosh Chauhan commented on HIVE-10567:
-

[~prasanth_j] Unfortunately partial partition spec partition(ds,hr) still needs 
to be specified if stats collection on all partitions is desired. We shall 
improve that syntax in a followup jira. 
In the meanwhile the bug specified here for partialscan is relevant and need to 
be fixed.

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-06-28 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067492#comment-16067492
 ] 

Hive QA commented on HIVE-10567:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12729689/HIVE-10567.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10829 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
 (batchId=238)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
 (batchId=238)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=238)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
 (batchId=146)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=233)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=233)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootstrapFunctionReplication
 (batchId=217)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionIncrementalReplication
 (batchId=217)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testCreateFunctionWithFunctionBinaryJarsOnHDFS
 (batchId=217)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
 (batchId=178)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
 (batchId=178)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation 
(batchId=178)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5819/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5819/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5819/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12729689 - PreCommit-HIVE-Build

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2017-06-28 Thread Prasanth Jayachandran (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067143#comment-16067143
 ] 

Prasanth Jayachandran commented on HIVE-10567:
--

[~ashutoshc] This is related to the feature where analyze does not need 
partition spec to be explicitly defined. right? If partition spec is missing, 
analyze will happen for all partitions. isn't it?

> partial scan for rcfile table doesn't work for dynamic partition
> 
>
> Key: HIVE-10567
> URL: https://issues.apache.org/jira/browse/HIVE-10567
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Processor
>Affects Versions: 0.14.0, 1.0.0
>Reporter: Thomas Friedrich
>Assignee: Thomas Friedrich
>Priority: Minor
>  Labels: rcfile
> Attachments: HIVE-10567.1.patch
>
>
> HIVE-3958 added support for partial scan for RCFile. This works fine for 
> static partitions (for example: analyze table analyze_srcpart_partial_scan 
> PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
> For dynamic partition, the analyze files with an IOException 
> "java.io.IOException: No input paths specified in job":
> hive> ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
> PARTIALSCAN;
> java.io.IOException: No input paths specified in job
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
> at 
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
> at 
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2015-04-30 Thread Thomas Friedrich (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522619#comment-14522619
 ] 

Thomas Friedrich commented on HIVE-10567:
-

Chaoyu, I attached a proposed patch. The problem is in method 
getInputPathsForPartialScan in class GenMapRedUtils.java. 
I added the case for DYNAMIC_PARTITION, but wasn't sure about the 
aggregrationKey. In the current patch, the aggregationKey is just the table 
name and the PartialScanMapper will join this with the task id which is 
different for each partition (one task per partition):
org.apache.hadoop.hive.ql.stats.fs.FSStatsPublisher: Writing stats in it : 
{default.testtable/00/={numRows=2, rawDataSize=16}}
org.apache.hadoop.hive.ql.stats.fs.FSStatsPublisher: Writing stats in it : 
{default.testtable/01/={numRows=1, rawDataSize=8}}
The output seems ok to me. 
Do you know whether the aggregationKey should be set to a different value, like 
in the STATIC_PARTITION case?

I would like to add a unit test for this case as well, that's why I didn't 
submit the patch yet.

 partial scan for rcfile table doesn't work for dynamic partition
 

 Key: HIVE-10567
 URL: https://issues.apache.org/jira/browse/HIVE-10567
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Affects Versions: 0.14.0, 1.0.0
Reporter: Thomas Friedrich
Assignee: Chaoyu Tang
Priority: Minor
  Labels: rcfile
 Attachments: HIVE-10567.1.patch


 HIVE-3958 added support for partial scan for RCFile. This works fine for 
 static partitions (for example: analyze table analyze_srcpart_partial_scan 
 PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
 For dynamic partition, the analyze files with an IOException 
 java.io.IOException: No input paths specified in job:
 hive ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
 PARTIALSCAN;
 java.io.IOException: No input paths specified in job
 at 
 org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
 at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-10567) partial scan for rcfile table doesn't work for dynamic partition

2015-04-30 Thread Chaoyu Tang (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522625#comment-14522625
 ] 

Chaoyu Tang commented on HIVE-10567:


Hi [~tfriedr], I have not started to look into this JIRA yet. Since you are 
working on this, please feel free to reassign it to yourself. Thanks

 partial scan for rcfile table doesn't work for dynamic partition
 

 Key: HIVE-10567
 URL: https://issues.apache.org/jira/browse/HIVE-10567
 Project: Hive
  Issue Type: Improvement
  Components: Query Processor
Affects Versions: 0.14.0, 1.0.0
Reporter: Thomas Friedrich
Assignee: Chaoyu Tang
Priority: Minor
  Labels: rcfile
 Attachments: HIVE-10567.1.patch


 HIVE-3958 added support for partial scan for RCFile. This works fine for 
 static partitions (for example: analyze table analyze_srcpart_partial_scan 
 PARTITION(ds='2008-04-08',hr=11) compute statistics partialscan).
 For dynamic partition, the analyze files with an IOException 
 java.io.IOException: No input paths specified in job:
 hive ANALYZE TABLE testtable PARTITION(col_varchar) COMPUTE STATISTICS 
 PARTIALSCAN;
 java.io.IOException: No input paths specified in job
 at 
 org.apache.hadoop.hive.ql.io.HiveInputFormat.getInputPaths(HiveInputFormat.java:318)
 at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:459)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:624)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:616)
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)