Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-22 Thread Andrew Sherman via Review Board 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > testutils/ptest2/conf/deployed/master-mr2.properties
> > Line 75 (original), 75 (patched)
> > 
> >
> > have you manually deployed these changes to the ptest server? this file 
> > is just a copy of whats already been deployed, so its just for reference
> > 
> > also, why skip batching?
> 
> Andrew Sherman wrote:
> OK I have no idea what ut.itests.qtest.skipBatching means I just copied 
> TestEncryptedHDFSCliDriver :-(
> 
> As for deploying these changes, I don't know what that means, my new 
> tests did appear to run. can you explain more?
> 
> Sahil Takiar wrote:
> anytime you add a new cli driver, you have to manually modify a file on 
> the ptest master server, you have to modify the file 
> `/usr/local/hiveptest/profiles/master-mr2.properties` you probably don't have 
> permissions though, so let me know the final diff for the this file and I can 
> deploy it.
> 
> can we check if batching can be used for this? i think batching means 
> that q-tests get bundled together into a "batch" of q-tests that are run in a 
> single `mvn test` command
> 
> Andrew Sherman wrote:
> Batching works locally for me so I have removed this. I'll provide a 
> patch for master-mr2.properties before commit

The weird thing is that the new driver does get run already, and my two new 
tests run: 
https://builds.apache.org/job/PreCommit-HIVE-Build/3/testReport/org.apache.hadoop.hive.cli/TestErasureCodingHDFSCliDriver/
Even so, please update the master-mr2.properties file, maybe there is some 
sublety. The diff in 
https://issues.apache.org/jira/secure/attachment/12924372/HIVE-18117.9.patch#file-16
 is good.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-16 Thread Sahil Takiar 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?
> 
> Andrew Sherman wrote:
> I think it is, as tez has its own hive-site.xml but I am not 100% sure
> 
> Sahil Takiar wrote:
> yes, tez has its own `hive-site.xml`, as does Spark; so do we need to do 
> this for Spark too?
> 
> Andrew Sherman wrote:
> That would make sense but then we have to decide if that is 
> data/conf/spark/local or data/conf/spark/local :-(
> These code paths are confusing becaus it is hard to anticipate how 
> they'll be used.

you can use `data/conf/spark/standalone` for `MiniClusterType.spark`; and 
`data/conf/spark/yarn-cluster` for `MiniClusterType.miniSparkOnYarn`. if u 
think this isn't worth adding support for, then i would suggest explicitly 
doing that rather than supporting certain execution engines and not others


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Andrew Sherman via Review Board 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?
> 
> Andrew Sherman wrote:
> I think it is, as tez has its own hive-site.xml but I am not 100% sure
> 
> Sahil Takiar wrote:
> yes, tez has its own `hive-site.xml`, as does Spark; so do we need to do 
> this for Spark too?

That would make sense but then we have to decide if that is 
data/conf/spark/local or data/conf/spark/local :-(
These code paths are confusing becaus it is hard to anticipate how they'll be 
used.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Sahil Takiar 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?
> 
> Andrew Sherman wrote:
> I think it is, as tez has its own hive-site.xml but I am not 100% sure

yes, tez has its own `hive-site.xml`, as does Spark; so do we need to do this 
for Spark too?


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Sahil Takiar 


> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?
> 
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my 
> ealrier reply)
> 
> Sahil Takiar wrote:
> why would you want to do that? whats the use case? shouldn't 
> `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm 
> not sure how you could run any EC commands against a local filesystem, 
> wouldn't they all be no-ops?
> 
> Andrew Sherman wrote:
> They would. When I'm developing an EC test I may sometimes want to run 
> the test in TestCLiDriver with erasure commands being no-ops as a way to 
> validate the script. Myabe this is weird, but I've been doing it.

I see, makes sense.


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Andrew Sherman via Review Board 


> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?
> 
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my 
> ealrier reply)
> 
> Sahil Takiar wrote:
> why would you want to do that? whats the use case? shouldn't 
> `TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm 
> not sure how you could run any EC commands against a local filesystem, 
> wouldn't they all be no-ops?

They would. When I'm developing an EC test I may sometimes want to run the test 
in TestCLiDriver with erasure commands being no-ops as a way to validate the 
script. Myabe this is weird, but I've been doing it.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Andrew Sherman via Review Board 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already
> 
> Sahil Takiar wrote:
> then is the tez specific code necessary?

I think it is, as tez has its own hive-site.xml but I am not 100% sure


> On May 10, 2018, noon, Sahil Takiar wrote:
> > testutils/ptest2/conf/deployed/master-mr2.properties
> > Line 75 (original), 75 (patched)
> > 
> >
> > have you manually deployed these changes to the ptest server? this file 
> > is just a copy of whats already been deployed, so its just for reference
> > 
> > also, why skip batching?
> 
> Andrew Sherman wrote:
> OK I have no idea what ut.itests.qtest.skipBatching means I just copied 
> TestEncryptedHDFSCliDriver :-(
> 
> As for deploying these changes, I don't know what that means, my new 
> tests did appear to run. can you explain more?
> 
> Sahil Takiar wrote:
> anytime you add a new cli driver, you have to manually modify a file on 
> the ptest master server, you have to modify the file 
> `/usr/local/hiveptest/profiles/master-mr2.properties` you probably don't have 
> permissions though, so let me know the final diff for the this file and I can 
> deploy it.
> 
> can we check if batching can be used for this? i think batching means 
> that q-tests get bundled together into a "batch" of q-tests that are run in a 
> single `mvn test` command

Batching works locally for me so I have removed this. I'll provide a patch for 
master-mr2.properties before commit


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Sahil Takiar 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
> looks like it does already

then is the tez specific code necessary?


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-15 Thread Sahil Takiar 


> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?
> 
> Andrew Sherman wrote:
> So you could run an ec test in TestCliDriver (sorry reviewboard lost my 
> ealrier reply)

why would you want to do that? whats the use case? shouldn't 
`TestErasureCodingHDFSCliDriver` encapsulate all EC-related tests? plus I'm not 
sure how you could run any EC commands against a local filesystem, wouldn't 
they all be no-ops?


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 89129f99fe63f0384aff965ad665770d11e9af04 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-14 Thread Andrew Sherman via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/
---

(Updated May 14, 2018, 9:44 p.m.)


Review request for hive and Sahil Takiar.


Repository: hive-git


Description
---

TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
"ErasureProcessor"
which allows .q files to contain Erasure Coding commands similar to those 
provided
by the hdfs ec command
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
The Erasure Coding functionality is exposed through a new shim 
"HdfsFileErasureCodingPolicy".
At this stage there are two .q files:
erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
via
TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
erasure_simple.q (which does some trivial queries to demonstrate basic 
functionality).
More tests will come in future commits.


Diffs (updated)
-

  
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
 PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
16571b3ff3288dfb99fbca570452592cc1650f9a 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
3d47991b603c94c8da2106e67192c8513ef783a7 
  ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
89129f99fe63f0384aff965ad665770d11e9af04 
  
ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
 de43c2866f64e2ed5c74eab450de28f1a79248dc 
  ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
  ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
  ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
PRE-CREATION 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
ec06a88dc21346473bec6589c703167d50e3b367 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
b89081761780bf1f305d0196bb94bb0b54f7184f 
  testutils/ptest2/conf/deployed/master-mr2.properties 
7edc307f85744d60d322ad8087164625677fc230 


Diff: https://reviews.apache.org/r/67023/diff/3/

Changes: https://reviews.apache.org/r/67023/diff/2-3/


Testing
---


Thanks,

Andrew Sherman

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-14 Thread Andrew Sherman via Review Board 


> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > itests/src/test/resources/testconfiguration.properties
> > Line 1693 (original), 1693 (patched)
> > 
> >
> > why would we want the ec commands to work outside the 
> > `TestErasureCodingHDFSCliDriver`?

So you could run an ec test in TestCliDriver (sorry reviewboard lost my ealrier 
reply)


> On May 14, 2018, 4:57 p.m., Sahil Takiar wrote:
> > ql/src/test/queries/clientpositive/erasure_commands.q
> > Lines 2 (patched)
> > 
> >
> > why would we want to run this on the local fs?

I'm imaginging wanting to run an ec test in TestCliDriver without changing it 
(sorry reviewboard lost my ealrier reply)


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---


On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 11, 2018, 11:38 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 6628336807b06cab49063673be0d8e9c5b5a7101 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-14 Thread Andrew Sherman via Review Board 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx
> 
> Sahil Takiar wrote:
> interesting, should we handle other cluster types like Spark or MR too?

looks like it does already


> On May 10, 2018, noon, Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
> > Lines 545 (patched)
> > 
> >
> > whats the cache for? can q-tests even specify custom URIs? whats the 
> > use case for support multiple fs URIs?
> 
> Andrew Sherman wrote:
> OK I admit I copied this code from the way hdfsEncryptionShims works 
> without fully understanding it.
> 
> Sahil Takiar wrote:
> can we delete it then? i didn't realize this would require modifying code 
> outside of itests, so I think we should make any changes to core Hive as 
> minimal as possible

deleted


> On May 10, 2018, noon, Sahil Takiar wrote:
> > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java
> > Lines 699 (patched)
> > 
> >
> > whats the use case for this dummy class? so we can run ec tests on 
> > hadoop versions that don't support ec? wouldn't be just disable the 
> > clidriver entirely for versions that don't support ec?
> 
> Andrew Sherman wrote:
> I'm imagining wanting to run a test on bith EC and non-EC
> 
> Sahil Takiar wrote:
> i thought the `NoopHdfsErasureCodingShim` was used when "the hadoop 
> version does not support hdfs Erasure Coding". u can still run on EC and 
> non-EC folders without this, right?

Yes that is the use of NoopHdfsErasureCodingShim. I think you always need 2 
implementations of the interface


> On May 10, 2018, noon, Sahil Takiar wrote:
> > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java
> > Lines 741 (patched)
> > 
> >
> > since we have a cache anyway, would it make more sense to just remove 
> > this and make it a loading cache?
> 
> Andrew Sherman wrote:
> I don't know, maybe. There is some advantage in keeping the same 
> structure as the HdfsEncryptionShim.
> 
> Sahil Takiar wrote:
> see comment above about removing the cache

OK I have removed the cache and it does make the code simpler


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 11, 2018, 11:38 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 6628336807b06cab49063673be0d8e9c5b5a7101 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-14 Thread Sahil Takiar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review203040
---




itests/src/test/resources/testconfiguration.properties
Line 1693 (original), 1693 (patched)


why would we want the ec commands to work outside the 
`TestErasureCodingHDFSCliDriver`?



ql/src/test/queries/clientpositive/erasure_commands.q
Lines 2 (patched)


why would we want to run this on the local fs?


- Sahil Takiar


On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 11, 2018, 11:38 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 6628336807b06cab49063673be0d8e9c5b5a7101 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-14 Thread Sahil Takiar 


> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > 
> >
> > is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
> Ha good question. Yes it is necessary as setClusterType() does not always 
> set the cluster type :-( - it allows the cluster type to overridden with 
> -Dclustermode=xxx

interesting, should we handle other cluster types like Spark or MR too?


> On May 10, 2018, noon, Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java
> > Lines 89 (patched)
> > 
> >
> > an `echo` command seems like a useful feature for all q-tests, does it 
> > need to be EC specific?
> 
> Andrew Sherman wrote:
> I put this in as I want it. I did think about putting it somewhere else 
> but there is no generic test-only CommandProcessor. Do you see an obvious 
> place to put it?

if its not easy to add for others, then just leave it


> On May 10, 2018, noon, Sahil Takiar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java
> > Lines 545 (patched)
> > 
> >
> > whats the cache for? can q-tests even specify custom URIs? whats the 
> > use case for support multiple fs URIs?
> 
> Andrew Sherman wrote:
> OK I admit I copied this code from the way hdfsEncryptionShims works 
> without fully understanding it.

can we delete it then? i didn't realize this would require modifying code 
outside of itests, so I think we should make any changes to core Hive as 
minimal as possible


> On May 10, 2018, noon, Sahil Takiar wrote:
> > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java
> > Lines 699 (patched)
> > 
> >
> > whats the use case for this dummy class? so we can run ec tests on 
> > hadoop versions that don't support ec? wouldn't be just disable the 
> > clidriver entirely for versions that don't support ec?
> 
> Andrew Sherman wrote:
> I'm imagining wanting to run a test on bith EC and non-EC

i thought the `NoopHdfsErasureCodingShim` was used when "the hadoop version 
does not support hdfs Erasure Coding". u can still run on EC and non-EC folders 
without this, right?


> On May 10, 2018, noon, Sahil Takiar wrote:
> > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java
> > Lines 741 (patched)
> > 
> >
> > since we have a cache anyway, would it make more sense to just remove 
> > this and make it a loading cache?
> 
> Andrew Sherman wrote:
> I don't know, maybe. There is some advantage in keeping the same 
> structure as the HdfsEncryptionShim.

see comment above about removing the cache


> On May 10, 2018, noon, Sahil Takiar wrote:
> > testutils/ptest2/conf/deployed/master-mr2.properties
> > Line 75 (original), 75 (patched)
> > 
> >
> > have you manually deployed these changes to the ptest server? this file 
> > is just a copy of whats already been deployed, so its just for reference
> > 
> > also, why skip batching?
> 
> Andrew Sherman wrote:
> OK I have no idea what ut.itests.qtest.skipBatching means I just copied 
> TestEncryptedHDFSCliDriver :-(
> 
> As for deploying these changes, I don't know what that means, my new 
> tests did appear to run. can you explain more?

anytime you add a new cli driver, you have to manually modify a file on the 
ptest master server, you have to modify the file 
`/usr/local/hiveptest/profiles/master-mr2.properties` you probably don't have 
permissions though, so let me know the final diff for the this file and I can 
deploy it.

can we check if batching can be used for this? i think batching means that 
q-tests get bundled together into a "batch" of q-tests that are run in a single 
`mvn test` command


- Sahil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
---


On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 11, 2018, 11:38 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-11 Thread Andrew Sherman via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202981
---




ql/src/test/queries/clientpositive/erasure_simple.q
Lines 33 (patched)


should be 'select key, value from src;'


- Andrew Sherman


On May 11, 2018, 11:38 p.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> ---
> 
> (Updated May 11, 2018, 11:38 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
> "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those 
> provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim 
> "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
> via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic 
> functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -
> 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 6628336807b06cab49063673be0d8e9c5b5a7101 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba 
>   
> ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
> 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
> 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
> 9f65a771f95a7c0bd3fdb4e56e47c0fc70235850 
>   
> ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
>  de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
> ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
> b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 
> 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.

2018-05-11 Thread Andrew Sherman via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/
---

(Updated May 11, 2018, 11:38 p.m.)


Review request for hive and Sahil Takiar.


Repository: hive-git


Description
---

TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor 
"ErasureProcessor"
which allows .q files to contain Erasure Coding commands similar to those 
provided
by the hdfs ec command
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
The Erasure Coding functionality is exposed through a new shim 
"HdfsFileErasureCodingPolicy".
At this stage there are two .q files:
erasure_commnds.q (a simple test to show ERASURE commands can run on local fs 
via
TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
erasure_simple.q (which does some trivial queries to demonstrate basic 
functionality).
More tests will come in future commits.


Diffs (updated)
-

  
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java
 PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
6628336807b06cab49063673be0d8e9c5b5a7101 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
750fc69c5f210ca8f7bfe81b82ee9e3001fc07ba 
  ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 
3d47991b603c94c8da2106e67192c8513ef783a7 
  ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 
56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
  ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 
9f65a771f95a7c0bd3fdb4e56e47c0fc70235850 
  
ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java
 de43c2866f64e2ed5c74eab450de28f1a79248dc 
  ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
  ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
  ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
PRE-CREATION 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 
ec06a88dc21346473bec6589c703167d50e3b367 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java 
b89081761780bf1f305d0196bb94bb0b54f7184f 
  testutils/ptest2/conf/deployed/master-mr2.properties 
7edc307f85744d60d322ad8087164625677fc230 


Diff: https://reviews.apache.org/r/67023/diff/2/

Changes: https://reviews.apache.org/r/67023/diff/1-2/


Testing
---


Thanks,

Andrew Sherman