[
https://issues.apache.org/jira/browse/HCATALOG-239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215869#comment-13215869
]
[email protected] commented on HCATALOG-239:
--------------------------------------------------------
bq. On 2012-02-24 07:07:57, Francis Liu wrote:
bq. > trunk/src/java/org/apache/hcatalog/data/LazyHCatRecord.java, line 106
bq. > <https://reviews.apache.org/r/3945/diff/2/?file=85617#file85617line106>
bq. >
bq. > Does it really ahve have to be throws exception? Can you be more
specific?
It winds up being SerDeException, but we do not want to leak knowledge of that
exception upstream. It could have been cast to IOException for clarity - that'd
probably be a better ctor.
bq. On 2012-02-24 07:07:57, Francis Liu wrote:
bq. > trunk/src/java/org/apache/hcatalog/mapreduce/HCatBaseInputFormat.java,
line 144
bq. > <https://reviews.apache.org/r/3945/diff/2/?file=85622#file85622line144>
bq. >
bq. > This doesn't look right, with this the current conf, partinfo will
get the location of the previous partinfo? You really should put this in
configureInput, that way partInto.jobProperties would've been set with the
correct input locaiton prior to this.
I'm not sure about moving it to another location, but Francis is right - you
need isolation of the confs being passed in to setInputPath. An easy way to do
this is to change the signature of setInputPath to JobConf, since you're
initializing a new one for each partition, and pass in the newly created
JobConf to it.
bq. On 2012-02-24 07:07:57, Francis Liu wrote:
bq. > trunk/src/java/org/apache/hcatalog/mapreduce/HCatBaseInputFormat.java,
line 171
bq. > <https://reviews.apache.org/r/3945/diff/2/?file=85622#file85622line171>
bq. >
bq. > jobConf must be copied back to JobContext.
Disagree. By the time getSplits is called, we have no more control over what
gets stored in JobContext. There's no value in storing it back into JobContext,
that info will be lost anyway.
- Sushanth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3945/#review5313
-----------------------------------------------------------
On 2012-02-24 03:32:30, Vikram Dixit Kumaraswamy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3945/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-02-24 03:32:30)
bq.
bq.
bq. Review request for Sushanth Sowmyan and Francis Liu.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Incorporated Francis' comments.
bq.
bq.
bq. This addresses bug HCATALOG-239.
bq. https://issues.apache.org/jira/browse/HCATALOG-239
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. trunk/src/java/org/apache/hcatalog/common/ErrorType.java 1292521
bq. trunk/src/java/org/apache/hcatalog/common/HCatUtil.java 1292521
bq. trunk/src/java/org/apache/hcatalog/data/DataType.java 1292521
bq.
trunk/src/java/org/apache/hcatalog/data/HCatRecordObjectInspectorFactory.java
1292521
bq. trunk/src/java/org/apache/hcatalog/data/LazyHCatRecord.java 1292521
bq.
trunk/src/java/org/apache/hcatalog/mapreduce/DefaultOutputCommitterContainer.java
1292521
bq.
trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputCommitterContainer.java
1292521
bq.
trunk/src/java/org/apache/hcatalog/mapreduce/FileOutputFormatContainer.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/HCatBaseInputFormat.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/HCatBaseOutputFormat.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/HCatOutputFormat.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/HCatRecordReader.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/HCatSplit.java 1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/HCatTableInfo.java 1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/InitializeInput.java
1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/InputJobInfo.java 1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/InternalUtil.java 1292521
bq. trunk/src/java/org/apache/hcatalog/mapreduce/PartInfo.java 1292521
bq. trunk/src/java/org/apache/hcatalog/pig/HCatLoader.java 1292521
bq. trunk/src/test/org/apache/hcatalog/data/TestHCatRecordSerDe.java 1292521
bq. trunk/src/test/org/apache/hcatalog/data/TestLazyHCatRecord.java 1292521
bq.
trunk/src/test/org/apache/hcatalog/mapreduce/TestHCatHiveCompatibility.java
1292521
bq. trunk/src/test/org/apache/hcatalog/pig/TestHCatLoaderComplexSchema.java
1292521
bq.
bq. Diff: https://reviews.apache.org/r/3945/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Vikram
bq.
bq.
> Changes to HCatInputFormat to make it use SerDes instead of StorageDrivers
> --------------------------------------------------------------------------
>
> Key: HCATALOG-239
> URL: https://issues.apache.org/jira/browse/HCATALOG-239
> Project: HCatalog
> Issue Type: Sub-task
> Components: storage handlers
> Affects Versions: 0.4
> Reporter: Alan Gates
> Assignee: Vikram Dixit K
> Fix For: 0.4
>
> Attachments: 239.patch, 239_1.patch, 239_2.patch, 239_3.patch,
> 239_4.patch, 239_5.patch, 239_6.patch, 239_7.patch
>
>
> This JIRA covers changes to HCatInputFormat and InputJobInfo. See
> HCATALOG-237 for details and design notes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
