[
https://issues.apache.org/jira/browse/PIG-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576168#action_12576168
]
Pi Song commented on PIG-114:
-----------------------------
Ben,
Sorry that I led to a long story again. I think whether to do marker interfaces
or not can be addressed when we implement optimization.
At this stage I just want to fix this bug which as discussed above will be
fixed by
(1) Add a structure to identify reversible property of StoreFunc
(2) Fix optimization engine not to do reverse if it's not safe
(3) Unit testing
I have implemented (1) using what I've proposed before
{noformat}
LoadFunc
StoreFunc
ReversibleLoadStoreFunc extends LoadFunc, StoreFunc
{noformat}
As you've said
{quote}
Alternatively we could add a method to StoreFunc:
LoadFunc getReloader();
{quote}
We still cannot conclude your way or my way which way is better. But since I've
already implemented using my way. Let's use my one. ( Meritocrazy :D )
For (2) I've fixed optimization in both local and mapreduce engines
For (3) This time I do an end-to-end test. I'm sorry for that. I find it much
more complex to do only plan compiler test (This involves both local and
mapreduce engines and physical ops are different). Also a usual way to do
optimization test is to compare actual output structure with expected output
structure (deterministic optimization engine). We still don't have that
framework yet. This should be done as a part of optimization work.
> store one alias/logicalPlan twice leads to instantiation of StoreFunc as
> LoadFunc
> ---------------------------------------------------------------------------------
>
> Key: PIG-114
> URL: https://issues.apache.org/jira/browse/PIG-114
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Johannes Zillmann
> Assignee: Pi Song
> Attachments: PIG114_FixOptimize_Sample.patch,
> pigPatch-storeTwice-620665.patch
>
>
> Calling PigServer#store() twice for an alias results in following exception :
> {noformat}
> java.lang.RuntimeException: java.lang.ClassCastException:
> org.apache.pig.test.DummyStoreFunc cannot be cast to org.apache.pig.LoadFunc
> at
> org.apache.pig.backend.local.executionengine.POLoad.<init>(POLoad.java:59)
> at
> org.apache.pig.backend.local.executionengine.LocalExecutionEngine.doCompile(LocalExecutionEngine.java:167)
> at
> org.apache.pig.backend.local.executionengine.LocalExecutionEngine.doCompile(LocalExecutionEngine.java:184)
> at
> org.apache.pig.backend.local.executionengine.LocalExecutionEngine.doCompile(LocalExecutionEngine.java:184)
> at
> org.apache.pig.backend.local.executionengine.LocalExecutionEngine.compile(LocalExecutionEngine.java:111)
> at
> org.apache.pig.backend.local.executionengine.LocalExecutionEngine.compile(LocalExecutionEngine.java:90)
> at
> org.apache.pig.backend.local.executionengine.LocalExecutionEngine.compile(LocalExecutionEngine.java:1)
> at org.apache.pig.PigServer.store(PigServer.java:330)
> at org.apache.pig.PigServer.store(PigServer.java:317)
> at org.apache.pig.test.StoreTwiceTest.testIt(StoreTwiceTest.java:31)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at junit.framework.TestCase.runTest(TestCase.java:164)
> at junit.framework.TestCase.runBare(TestCase.java:130)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:120)
> at junit.framework.TestSuite.runTest(TestSuite.java:228)
> at junit.framework.TestSuite.run(TestSuite.java:223)
> at
> org.junit.internal.runners.OldTestClassRunner.run(OldTestClassRunner.java:35)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:45)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:460)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:673)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:386)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:196)
> Caused by: java.lang.ClassCastException: org.apache.pig.test.DummyStoreFunc
> cannot be cast to org.apache.pig.LoadFunc
> at
> org.apache.pig.backend.local.executionengine.POLoad.<init>(POLoad.java:57)
> ... 28 more
> {noformat}
> I will attach a patch with a test scenario for this. Basically the code is as
> follow:
> {noformat}PigServer pig = new PigServer(ExecType.LOCAL);
> pig
> .registerQuery("A = LOAD
> 'test/org/apache/pig/test/StoreTwiceTest.java' USING "
> + DummyLoadFunc.class.getName() + "();");
> pig.registerQuery("B = FOREACH A GENERATE * ;");
> File outputFile = new File("/tmp/testPigOutput");
> outputFile.delete();
> pig.store("A", outputFile.getAbsolutePath(), DummyStoreFunc.class
> .getName()
> + "()");
> outputFile.delete();
> pig.store("B", outputFile.getAbsolutePath(), DummyStoreFunc.class
> .getName()
> + "()");
> outputFile.delete();
> assertEquals(2, _storedTuples.size());
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
