[jira] [Commented] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514948#comment-15514948
]
Kevin Ratnasekera commented on GORA-401:
Thank you [~alfonso.nishikawa] for reviewing and testing the patch :)
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.7
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 0h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GORA-412) Consider location of @SuppressWarnings("all") in compiled classes
[
https://issues.apache.org/jira/browse/GORA-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514939#comment-15514939
]
Kevin Ratnasekera commented on GORA-412:
Thanks [~alfonso.nishikawa] and [~lewismc]
> Consider location of @SuppressWarnings("all") in compiled classes
> -
>
> Key: GORA-412
> URL: https://issues.apache.org/jira/browse/GORA-412
> Project: Apache Gora
> Issue Type: Improvement
> Components: gora-compiler
>Affects Versions: 0.6
>Reporter: Lewis John McGibbney
>Assignee: Kevin Ratnasekera
> Fix For: 0.7
>
>
> Right now we silence Javac to any potential warnings by adding the following
> to compiled classes
> {code}
> /**
> * Autogenerated by Avro
> *
> * DO NOT EDIT DIRECTLY
> */
> package org.apache.gora;
> @SuppressWarnings("all")
> ...
> {code}
> This means that the Javadoc associated with the generated class is not
> interpreted correctly by clients such as Eclipse.
> I propose to either
> * remove the SupressWarnings altogether, or
> * have it generated underneath the Javadoc string.
> Any thoughts folks?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514877#comment-15514877
]
Hudson commented on GORA-401:
-
FAILURE: Integrated in Jenkins build gora-trunk #1689 (See
[https://builds.apache.org/job/gora-trunk/1689/])
adding fix for GORA-401 (kevin: rev ac6c42b37385fd05fa9776e53d6de41320a202c4)
* (edit)
gora-core/src/main/java/org/apache/gora/mapreduce/PersistentDeserializer.java
* (edit)
gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java
* (edit) gora-core/src/test/java/org/apache/gora/util/TestIOUtils.java
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.7
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 0h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
gora-trunk - Build # 1689 - Still Failing
The Apache Jenkins build system has built gora-trunk (build #1689) Status: Still Failing Check console output at https://builds.apache.org/job/gora-trunk/1689/ to view the results.
[jira] [Resolved] (GORA-370) Implement MapReduce 2.X tests
[ https://issues.apache.org/jira/browse/GORA-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney resolved GORA-370. --- Resolution: Fixed Resolved for some time now. > Implement MapReduce 2.X tests > - > > Key: GORA-370 > URL: https://issues.apache.org/jira/browse/GORA-370 > Project: Apache Gora > Issue Type: Test > Components: build process, gora-core, testing >Affects Versions: 0.5 >Reporter: Lewis John McGibbney >Assignee: Henry Saputra > Fix For: 0.7 > > > After committing GORA-346 it turns out that using Hadoop 2.X breaks our > lovely TestDataFileAvroStoreMapReduce test with the following > Error Message > org/apache/hadoop/hdfs/DistributedFileSystem > Stacktrace > java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/DistributedFileSystem > at java.lang.Class.getDeclaredMethods0(Native Method) > at java.lang.Class.privateGetDeclaredMethods(Class.java:2521) > at java.lang.Class.getMethod0(Class.java:2764) > at java.lang.Class.getMethod(Class.java:1653) > at > org.junit.internal.builders.SuiteMethodBuilder.hasSuiteMethod(SuiteMethodBuilder.java:20) > at > org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:13) > at > org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57) > at > org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29) > at > org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57) > at > org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hdfs.DistributedFileSystem > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 22 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GORA-370) Implement MapReduce 2.X tests
[ https://issues.apache.org/jira/browse/GORA-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated GORA-370: -- Fix Version/s: (was: 0.8) 0.7 > Implement MapReduce 2.X tests > - > > Key: GORA-370 > URL: https://issues.apache.org/jira/browse/GORA-370 > Project: Apache Gora > Issue Type: Test > Components: build process, gora-core, testing >Affects Versions: 0.5 >Reporter: Lewis John McGibbney >Assignee: Henry Saputra > Fix For: 0.7 > > > After committing GORA-346 it turns out that using Hadoop 2.X breaks our > lovely TestDataFileAvroStoreMapReduce test with the following > Error Message > org/apache/hadoop/hdfs/DistributedFileSystem > Stacktrace > java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/DistributedFileSystem > at java.lang.Class.getDeclaredMethods0(Native Method) > at java.lang.Class.privateGetDeclaredMethods(Class.java:2521) > at java.lang.Class.getMethod0(Class.java:2764) > at java.lang.Class.getMethod(Class.java:1653) > at > org.junit.internal.builders.SuiteMethodBuilder.hasSuiteMethod(SuiteMethodBuilder.java:20) > at > org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:13) > at > org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57) > at > org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29) > at > org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57) > at > org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hdfs.DistributedFileSystem > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 22 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GORA-316) Upgrade to Avro 1.8.x in gora-dynamodb
[ https://issues.apache.org/jira/browse/GORA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney resolved GORA-316. --- Resolution: Fixed https://github.com/apache/gora/blob/master/pom.xml#L683 > Upgrade to Avro 1.8.x in gora-dynamodb > -- > > Key: GORA-316 > URL: https://issues.apache.org/jira/browse/GORA-316 > Project: Apache Gora > Issue Type: Sub-task > Components: build process, storage >Reporter: Renato Javier Marroquín Mogrovejo >Assignee: Renato Javier Marroquín Mogrovejo > Fix For: 0.7 > > Attachments: GORA-316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GORA-316) Upgrade to Avro 1.7.x in gora-dynamodb
[ https://issues.apache.org/jira/browse/GORA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated GORA-316: -- Fix Version/s: (was: 0.8) 0.7 > Upgrade to Avro 1.7.x in gora-dynamodb > -- > > Key: GORA-316 > URL: https://issues.apache.org/jira/browse/GORA-316 > Project: Apache Gora > Issue Type: Sub-task > Components: build process, storage >Reporter: Renato Javier Marroquín Mogrovejo >Assignee: Renato Javier Marroquín Mogrovejo > Fix For: 0.7 > > Attachments: GORA-316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GORA-483) CI build for Gora
[ https://issues.apache.org/jira/browse/GORA-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated GORA-483: -- Fix Version/s: (was: 0.8) 0.7 > CI build for Gora > - > > Key: GORA-483 > URL: https://issues.apache.org/jira/browse/GORA-483 > Project: Apache Gora > Issue Type: Improvement > Components: build process >Reporter: Lewis John McGibbney >Assignee: Lewis John McGibbney > Fix For: 0.7 > > > Right now we could make code review an easier process by creating a TravisCI > build for Gora. It would tell us if the patch passes tests, etc. and aids the > peer review process for committers and contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GORA-316) Upgrade to Avro 1.8.x in gora-dynamodb
[ https://issues.apache.org/jira/browse/GORA-316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated GORA-316: -- Summary: Upgrade to Avro 1.8.x in gora-dynamodb (was: Upgrade to Avro 1.7.x in gora-dynamodb) > Upgrade to Avro 1.8.x in gora-dynamodb > -- > > Key: GORA-316 > URL: https://issues.apache.org/jira/browse/GORA-316 > Project: Apache Gora > Issue Type: Sub-task > Components: build process, storage >Reporter: Renato Javier Marroquín Mogrovejo >Assignee: Renato Javier Marroquín Mogrovejo > Fix For: 0.7 > > Attachments: GORA-316.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (GORA-483) CI build for Gora
[ https://issues.apache.org/jira/browse/GORA-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney resolved GORA-483. --- Resolution: Fixed > CI build for Gora > - > > Key: GORA-483 > URL: https://issues.apache.org/jira/browse/GORA-483 > Project: Apache Gora > Issue Type: Improvement > Components: build process >Reporter: Lewis John McGibbney >Assignee: Lewis John McGibbney > Fix For: 0.7 > > > Right now we could make code review an easier process by creating a TravisCI > build for Gora. It would tell us if the patch passes tests, etc. and aids the > peer review process for committers and contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GORA-483) CI build for Gora
[ https://issues.apache.org/jira/browse/GORA-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lewis John McGibbney updated GORA-483: -- Summary: CI build for Gora (was: TravisCI build for Gora) > CI build for Gora > - > > Key: GORA-483 > URL: https://issues.apache.org/jira/browse/GORA-483 > Project: Apache Gora > Issue Type: Improvement > Components: build process >Reporter: Lewis John McGibbney >Assignee: Lewis John McGibbney > Fix For: 0.8 > > > Right now we could make code review an easier process by creating a TravisCI > build for Gora. It would tell us if the patch passes tests, etc. and aids the > peer review process for committers and contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GORA-483) CI build for Gora
[ https://issues.apache.org/jira/browse/GORA-483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514800#comment-15514800 ] Lewis John McGibbney commented on GORA-483: --- I configured this within our Jenkins job. Now the Github issues should be automatically built against whenever a new PR comes into Github. > CI build for Gora > - > > Key: GORA-483 > URL: https://issues.apache.org/jira/browse/GORA-483 > Project: Apache Gora > Issue Type: Improvement > Components: build process >Reporter: Lewis John McGibbney >Assignee: Lewis John McGibbney > Fix For: 0.8 > > > Right now we could make code review an easier process by creating a TravisCI > build for Gora. It would tell us if the patch passes tests, etc. and aids the > peer review process for committers and contributors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated GORA-401:
--
Fix Version/s: (was: 0.8)
0.7
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.7
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 0h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alfonso Nishikawa updated GORA-401:
---
Remaining Estimate: 0h (was: 14h)
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.8
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 0h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514774#comment-15514774
]
Alfonso Nishikawa commented on GORA-401:
After GORA-495 everything seems to work.
Commited.
Thanks, [~djkevincr]!
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.8
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 14h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alfonso Nishikawa resolved GORA-401.
Resolution: Fixed
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.8
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 14h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GORA-401) Serialization and deserialization of Persistent does not hold the entity dirty state from Map to Reduce
[
https://issues.apache.org/jira/browse/GORA-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514771#comment-15514771
]
ASF GitHub Bot commented on GORA-401:
-
Github user asfgit closed the pull request at:
https://github.com/apache/gora/pull/89
> Serialization and deserialization of Persistent does not hold the entity
> dirty state from Map to Reduce
> ---
>
> Key: GORA-401
> URL: https://issues.apache.org/jira/browse/GORA-401
> Project: Apache Gora
> Issue Type: Bug
> Components: gora-core
>Affects Versions: 0.4, 0.5
> Environment: Tested on gora-0.4, but seems logically to hold on
> gora-0.5. HBase backend.
>Reporter: Alfonso Nishikawa
>Assignee: Kevin Ratnasekera
>Priority: Critical
> Labels: serialization
> Fix For: 0.8
>
> Attachments: GORA-401-tests.patch, GORA-401v1.patch,
> GORA-401v2.patch, GORA-401v3.patch, GORA-401v4.patch, GORA-401v5.patch
>
> Original Estimate: 35h
> Time Spent: 21h
> Remaining Estimate: 14h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialized.
> In GORA-321
> {{[PersistentSerializer|https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializer.java]}}
> went from using
> {{[PersistentDatumWriter|https://github.com/apache/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/PersistentDatumWriter.java](/Reader)}}
> to Avro's {{SpecificDatumWriter}}, delegating the serialization of the dirty
> field to Avro (but really not desirable to have that field as a main field in
> the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which
> will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it's
> phases, serializes entities (from Map to Reduce), and when deserializes finds
> all fields as "dirty", independently of what fields were modified in the Map,
> and overwrite all data in datastore (deleting much things: downloaded
> content, parsed content, etc).
> This effect can be seen in
> {{TestPersistentSerialization#testSerderEmployeeTwoFields}}, when debuging in
> {{TestIOUtils#testSerializeDeserialize}}. Proper breakpoints an inspections
> shows that, entities are "equal" when it's fields are equal. This is fine as
> "equal" definition, but another test must be added to check that
> serialization an deserialization keeps the dirty state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] gora pull request #89: adding fix for GORA-401
Github user asfgit closed the pull request at: https://github.com/apache/gora/pull/89 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[CassandraStore Tests] attempted to delete non-existing file...
Hi. Jenkis [1] fails because of this: "attempted to delete non-existing file system-compaction_history-tmp-jb-533-Statistics.db" Anyone know how to fix it? I know it does not means it is actually failing to build, but I guess it will be good to have clean builds. Priority is "ultra low", just to know if someone has some intuition about what is happening :) Regards, Alfonso [1] - https://builds.apache.org/job/gora-trunk/org.apache.gora$gora-cassandra/1688/testReport/junit/org.apache.gora.cassandra.store/TestCassandraStore/org_apache_gora_cassandra_store_TestCassandraStore/
[jira] [Commented] (GORA-412) Consider location of @SuppressWarnings("all") in compiled classes
[
https://issues.apache.org/jira/browse/GORA-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514359#comment-15514359
]
Hudson commented on GORA-412:
-
FAILURE: Integrated in Jenkins build gora-trunk #1688 (See
[https://builds.apache.org/job/gora-trunk/1688/])
GORA-412 removed @SuppressWarnings(all) from record.vm velocity template
(kevin: rev 4cb5a8b4d46e9fc76cd91728855a431ddad2ca34)
* (edit)
gora-compiler/src/main/velocity/org/apache/gora/compiler/templates/record.vm
* (edit)
gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/Pageview.java
* (edit)
gora-core/src/examples/java/org/apache/gora/examples/generated/ImmutableFields.java
* (edit) gora-goraci/src/main/java/org/apache/gora/goraci/generated/CINode.java
* (edit)
gora-core/src/examples/java/org/apache/gora/examples/generated/Metadata.java
* (edit)
gora-core/src/examples/java/org/apache/gora/examples/generated/WebPage.java
* (edit) gora-core/src/examples/java/org/apache/gora/examples/generated/V2.java
* (edit)
gora-core/src/examples/java/org/apache/gora/examples/generated/TokenDatum.java
* (edit)
gora-tutorial/src/main/java/org/apache/gora/tutorial/log/generated/MetricDatum.java
* (edit)
gora-core/src/examples/java/org/apache/gora/examples/generated/Employee.java
* (edit) gora-goraci/src/main/java/org/apache/gora/goraci/generated/Flushed.java
> Consider location of @SuppressWarnings("all") in compiled classes
> -
>
> Key: GORA-412
> URL: https://issues.apache.org/jira/browse/GORA-412
> Project: Apache Gora
> Issue Type: Improvement
> Components: gora-compiler
>Affects Versions: 0.6
>Reporter: Lewis John McGibbney
>Assignee: Kevin Ratnasekera
> Fix For: 0.7
>
>
> Right now we silence Javac to any potential warnings by adding the following
> to compiled classes
> {code}
> /**
> * Autogenerated by Avro
> *
> * DO NOT EDIT DIRECTLY
> */
> package org.apache.gora;
> @SuppressWarnings("all")
> ...
> {code}
> This means that the Javadoc associated with the generated class is not
> interpreted correctly by clients such as Eclipse.
> I propose to either
> * remove the SupressWarnings altogether, or
> * have it generated underneath the Javadoc string.
> Any thoughts folks?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (GORA-495) Gora compiler generates uncompilable databeans
[ https://issues.apache.org/jira/browse/GORA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alfonso Nishikawa resolved GORA-495. Resolution: Fixed > Gora compiler generates uncompilable databeans > -- > > Key: GORA-495 > URL: https://issues.apache.org/jira/browse/GORA-495 > Project: Apache Gora > Issue Type: Bug > Components: gora-compiler >Affects Versions: 0.6 >Reporter: Kevin Ratnasekera >Assignee: Kevin Ratnasekera > Fix For: 0.7 > > > This is due to 2 missing variable declarations in record velocity template. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GORA-495) Gora compiler generates uncompilable databeans
[ https://issues.apache.org/jira/browse/GORA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514253#comment-15514253 ] Alfonso Nishikawa commented on GORA-495: Commited. Many thanks, [~djkevincr] > Gora compiler generates uncompilable databeans > -- > > Key: GORA-495 > URL: https://issues.apache.org/jira/browse/GORA-495 > Project: Apache Gora > Issue Type: Bug > Components: gora-compiler >Affects Versions: 0.6 >Reporter: Kevin Ratnasekera >Assignee: Kevin Ratnasekera > Fix For: 0.7 > > > This is due to 2 missing variable declarations in record velocity template. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GORA-495) Gora compiler generates uncompilable databeans
[ https://issues.apache.org/jira/browse/GORA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514247#comment-15514247 ] ASF GitHub Bot commented on GORA-495: - Github user asfgit closed the pull request at: https://github.com/apache/gora/pull/88 > Gora compiler generates uncompilable databeans > -- > > Key: GORA-495 > URL: https://issues.apache.org/jira/browse/GORA-495 > Project: Apache Gora > Issue Type: Bug > Components: gora-compiler >Affects Versions: 0.6 >Reporter: Kevin Ratnasekera >Assignee: Kevin Ratnasekera > Fix For: 0.7 > > > This is due to 2 missing variable declarations in record velocity template. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] gora pull request #88: GORA-495 fix
Github user asfgit closed the pull request at: https://github.com/apache/gora/pull/88 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (GORA-495) Gora compiler generates uncompilable databeans
[ https://issues.apache.org/jira/browse/GORA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512630#comment-15512630 ] ASF GitHub Bot commented on GORA-495: - Github user alfonsonishikawa commented on the issue: https://github.com/apache/gora/pull/88 Didn't know about that "magic" :) This evening I will push it :D Thanks! > Gora compiler generates uncompilable databeans > -- > > Key: GORA-495 > URL: https://issues.apache.org/jira/browse/GORA-495 > Project: Apache Gora > Issue Type: Bug > Components: gora-compiler >Affects Versions: 0.6 >Reporter: Kevin Ratnasekera >Assignee: Kevin Ratnasekera > Fix For: 0.7 > > > This is due to 2 missing variable declarations in record velocity template. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] gora issue #88: GORA-495 fix
Github user alfonsonishikawa commented on the issue: https://github.com/apache/gora/pull/88 Didn't know about that "magic" :) This evening I will push it :D Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
