-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52782/
-----------------------------------------------------------

Review request for oozie, Attila Sasvari, Peter Cseh, Peter Bacsko, and Robert 
Kanter.


Repository: oozie-git


Description
-------

We get each 1000 rows into a separate JPA `EntityTransaction` to reduce heap 
size. Furthermore, of at least one row inside that tx fails, we retry the whole 
batch into separate `EntityTransaction`s each.

Following error handling is implemented:

1. check if all necessary tables are present and empty
2. rows are imported till the end even if there are skipped rows in the 
meanwhile
3. if at least one row is skipped in the meanwhile for some 
`ConstraintViolationException`, we delete all rows of all necessary tables. 
That enables the user to have the log messages of all the erroneous rows in one 
run, and Oozie database is never in an inconsistent state of some rows present, 
some not present of an import


Diffs
-----

  core/src/main/java/org/apache/oozie/service/JPAService.java 028381d 
  tools/src/main/java/org/apache/oozie/tools/OozieDBImportCLI.java 0e14a30 
  tools/src/test/java/org/apache/oozie/tools/TestDBLoadDump.java c43223e 
  tools/src/test/resources/dumpData/invalid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_slasum.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/invalid/ooziedb_wf.json PRE-CREATION 
  tools/src/test/resources/dumpData/ooziedb_ac.json 22bbdc2 
  tools/src/test/resources/dumpData/ooziedb_bna.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_bnj.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_ca.json 2715b94 
  tools/src/test/resources/dumpData/ooziedb_cj.json 979c10e 
  tools/src/test/resources/dumpData/ooziedb_slareg.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_slasum.json e69de29 
  tools/src/test/resources/dumpData/ooziedb_sysinfo.json 15de009 
  tools/src/test/resources/dumpData/ooziedb_wf.json 05e7e36 
  tools/src/test/resources/dumpData/valid/ooziedb_ac.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bna.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_bnj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_ca.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_cj.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slareg.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_slasum.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_sysinfo.json PRE-CREATION 
  tools/src/test/resources/dumpData/valid/ooziedb_wf.json PRE-CREATION 

Diff: https://reviews.apache.org/r/52782/diff/


Testing
-------

See `TestDBLoadDump` for further reference.


Thanks,

András Piros

Reply via email to