----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52782/#review180420 -----------------------------------------------------------
tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java Lines 204-208 (original), 298-317 (patched) <https://reviews.apache.org/r/52782/#comment255613> By reading in all the entries for one table into a list first then persisting it in oozie.db.import.batch.size batches the OOME won't go away. The oozie.db.import.batch.size should be taken into account while reading in the json file so we don't have to keep more than a oozie.db.import.batch.size entities in the memory at once. - Peter Cseh On July 4, 2017, noon, András Piros wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52782/ > ----------------------------------------------------------- > > (Updated July 4, 2017, noon) > > > Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and > Robert Kanter. > > > Repository: oozie-git > > > Description > ------- > > We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap > size. Furthermore, of at least one row inside that tx fails, we retry the > whole batch into separate `EntityTransaction`s each. > > Following error handling is implemented: > > 1. check if all necessary tables are present and empty > 2. rows are imported till the end even if there are skipped rows in the > meanwhile > 3. if at least one row is skipped in the meanwhile for some > `ConstraintViolationException`, we delete all rows of all necessary tables. > That enables the user to have the log messages of all the erroneous rows in > one run, and Oozie database is never in an inconsistent state of some rows > present, some not present of an import > > > Diffs > ----- > > core/src/main/resources/oozie-default.xml > c60a4581a84d4c67a1ac1cf3dfdc252b85ccd01c > tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java > 0e14a30693a76b8b2bdc2f7ceaf3f045d69f4155 > tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java > c43223ef05aa702be49565ba2626314628e63749 > tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION > tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION > tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION > tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION > tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION > tools/src/test/resources/dumpData/ooziedb_ac.json > tools/src/test/resources/dumpData/ooziedb_bna.json > tools/src/test/resources/dumpData/ooziedb_bnj.json > tools/src/test/resources/dumpData/ooziedb_ca.json > tools/src/test/resources/dumpData/ooziedb_cj.json > tools/src/test/resources/dumpData/ooziedb_slareg.json > tools/src/test/resources/dumpData/ooziedb_slasum.json > tools/src/test/resources/dumpData/ooziedb_sysinfo.json > tools/src/test/resources/dumpData/ooziedb_wf.json > tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION > tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION > tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION > tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/52782/diff/5/ > > > Testing > ------- > > See `TestDBLoadDump` for further reference. > > > Thanks, > > András Piros > >
