[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041188#comment-14041188
]
David Chen commented on HIVE-7094:
--
Thanks, Carl and Sushanth!
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Fix For: 0.14.0
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14037893#comment-14037893
]
Carl Steinbach commented on HIVE-7094:
--
[~sushanth]: I'm planning to commit this patch tonight. Please let me know if I
should hold off. Thanks.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036243#comment-14036243
]
Sushanth Sowmyan commented on HIVE-7094:
Hi David,
I've looked at the patch, and I think you're good, but could you please update
the reviewboard link with the latest patch you've uploaded so I can quickly
verify?
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036250#comment-14036250
]
David Chen commented on HIVE-7094:
--
Thanks, Sushanth. I have updated the RB with my latest patch.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036734#comment-14036734
]
Hive QA commented on HIVE-7094:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650889/HIVE-7094.5.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 5654 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/506/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/506/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-506/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650889
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036769#comment-14036769
]
David Chen commented on HIVE-7094:
--
The above test failures are unrelated to this patch. I think this patch should
be ready to be committed.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034418#comment-14034418
]
David Chen commented on HIVE-7094:
--
The Hive CLI and QL test failures are unrelated to my patch.
The only one that is concerning to me is the TestOrcHCatLoader failure.
However, I am not hitting that failure when I run the tests on my machine, but
I have rebased my changes on the current trunk. Nonetheless, all of the
HCatalog Core tests pass.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034436#comment-14034436
]
Sushanth Sowmyan commented on HIVE-7094:
Some of the HCat tests are flaky when run on the ptest framework (they succeed
when run on a serial run) - that's on my tofix radar. But in the meanwhile,
I'll try running the HCat tests with David's patch to verify.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034531#comment-14034531
]
Sushanth Sowmyan commented on HIVE-7094:
The only test that fails in my environment with this patch is
TestHCatPartitionPublish, and that's flaky in how it fails - it succeeds when
run by itself, as opposed to after other tests run, and that has nothing to do
with this patch. All other tests under hcatalog succeed.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14034544#comment-14034544
]
David Chen commented on HIVE-7094:
--
Thanks for your help, [~sushanth]! I have a separate ticket open to track down
the flakiness in TestHCatPartitionPublish (HIVE-7110).
Is there any more feedback before this patch can be committed?
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch,
HIVE-7094.5.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032749#comment-14032749
]
David Chen commented on HIVE-7094:
--
Previous patch was not created correctly. I have attached a new patch.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033325#comment-14033325
]
Hive QA commented on HIVE-7094:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650616/HIVE-7094.4.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 5611 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_columnar
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatLoader.testReadDataPrimitiveTypes
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/486/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/486/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-486/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650616
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch, HIVE-7094.4.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031977#comment-14031977
]
David Chen commented on HIVE-7094:
--
See HIVE-7230 for the patch for adding the Eclipse formatter file.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032064#comment-14032064
]
Hive QA commented on HIVE-7094:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12650393/HIVE-7094.3.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/478/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/478/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-478/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-478/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'conf/hive-default.xml.template'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
Reverted 'ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitPack.java'
Reverted
'ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestNewIntegerEncoding.java'
Reverted
'ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestSerializationUtils.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReader.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerReaderV2.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriter.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthIntegerWriterV2.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/SerializationUtils.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target
itests/custom-serde/target itests/util/target hcatalog/target
hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hwi/target
common/target common/src/gen service/target contrib/target serde/target
beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestUnrolledBitPack.java
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1602783.
At revision 1602783.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12650393
Separate out static/dynamic
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031188#comment-14031188
]
Carl Steinbach commented on HIVE-7094:
--
[~davidzchen]: +1. Can you please attach a new version of the patch to trigger
testing? If everything passes I will commit.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14031197#comment-14031197
]
David Chen commented on HIVE-7094:
--
Thanks, [~cwsteinbach]! I have addressed the remaining formatting issues using
the Eclipse formatter and uploaded a new patch.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch, HIVE-7094.3.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14030124#comment-14030124
]
David Chen commented on HIVE-7094:
--
Thanks for taking a look, [~cwsteinbach]. I have updated the RB with a new
revision.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028220#comment-14028220
]
David Chen commented on HIVE-7094:
--
Does someone have a chance to take a look at this?
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Components: HCatalog
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14015852#comment-14015852
]
David Chen commented on HIVE-7094:
--
This test failure appears to be unrelated to my patch. It is one of the
hcatalog-pig-adapter tests (which come after the hcatalog-core tests) and I had
observed the same test failure without this patch when testing HIVE-7110.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016143#comment-14016143
]
David Chen commented on HIVE-7094:
--
The above test failures are due to HIVE-7085 and do not appear to be related to
this patch.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016200#comment-14016200
]
Carl Steinbach commented on HIVE-7094:
--
[~davidzchen]: Is this ready for review now?
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016224#comment-14016224
]
David Chen commented on HIVE-7094:
--
[~cwsteinbach] Yes, this is ready for review.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014905#comment-14014905
]
Hive QA commented on HIVE-7094:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12647427/HIVE-7094.1.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 5571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_load_dyn_part1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_scriptfile1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_schema_evolution
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
org.apache.hadoop.hive.ql.exec.tez.TestTezTask.testSubmit
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimal
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalX
org.apache.hive.hcatalog.pig.TestOrcHCatPigStorer.testWriteDecimalXY
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/352/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/352/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-352/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12647427
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012780#comment-14012780
]
David Chen commented on HIVE-7094:
--
Posted patch for pre-commit testing.
RB: https://reviews.apache.org/r/22033/
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013214#comment-14013214
]
Carl Steinbach commented on HIVE-7094:
--
[~davidzchen]: Is this patch ready for review?
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7094) Separate out static/dynamic partitioning code in FileRecordWriterContainer
[
https://issues.apache.org/jira/browse/HIVE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013219#comment-14013219
]
David Chen commented on HIVE-7094:
--
Almost. I would like to fix HIVE-7110 first so that I can run this patch
through all the hcatalog-core tests and check that they pass.
Looks like pre-commit tests do not get run unless the ticket is in the Patch
Available state. I'll change to Patch Available for now and then post a comment
later when I am ready to have this reviewed.
Separate out static/dynamic partitioning code in FileRecordWriterContainer
--
Key: HIVE-7094
URL: https://issues.apache.org/jira/browse/HIVE-7094
Project: Hive
Issue Type: Sub-task
Reporter: David Chen
Assignee: David Chen
Attachments: HIVE-7094.1.patch
There are two major places in FileRecordWriterContainer that have the {{if
(dynamicPartitioning)}} condition: the constructor and write().
This is the approach that I am taking:
# Move the DP and SP code into two subclasses:
DynamicFileRecordWriterContainer and StaticFileRecordWriterContainer.
# Make FileRecordWriterContainer an abstract class that contains the common
code for both implementations. For write(), FileRecordWriterContainer will
call an abstract method that will provide the local RecordWriter,
ObjectInspector, SerDe, and OutputJobInfo.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
