Merge branch 'master' into S2GRAPH-180 * master: [S2GRAPH-186]: fix wrong escaping of double quotation marks fix json parse error fix wrong escaping of double quotation marks [S2GRAPH-184]: spark driver exit abnormally in the loader of s2jobs fix spark driver exit abnormally bug test s2jobs/loader.py on local. - add s2jobs subproject. - migrate bulk loader with spark 2.3.0. - add test cases for bulk loader. add integrate test on test case. remove unnecessary HConnection on HBaseContext and add generateHFile test case. bug fix on vertex property parsing. add test case for TransferHFile. add Vertex upsert on schemaVersion v4. fix null pointer error on Bulk Loader.
Project: http://git-wip-us.apache.org/repos/asf/incubator-s2graph/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-s2graph/commit/ca7adee6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-s2graph/tree/ca7adee6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-s2graph/diff/ca7adee6 Branch: refs/heads/master Commit: ca7adee619d577c4b7938471f89fc308f2ff8ff6 Parents: 823a884 0520ffc Author: daewon <[email protected]> Authored: Mon Mar 26 15:55:07 2018 +0900 Committer: daewon <[email protected]> Committed: Mon Mar 26 15:55:07 2018 +0900 ---------------------------------------------------------------------- CHANGES | 5 + build.sbt | 11 +- loader/build.sbt | 5 +- loader/loader.py | 85 +- .../s2graph/loader/spark/HBaseContext.scala | 12 +- .../loader/spark/HBaseRDDFunctions.scala | 3 +- .../loader/subscriber/GraphSubscriber.scala | 10 + .../loader/subscriber/TransferToHFile.scala | 278 ++++-- .../loader/subscriber/TransferToHFileTest.scala | 232 +++++ project/Common.scala | 1 + .../s2graph/core/GraphElementBuilder.scala | 2 +- .../org/apache/s2graph/core/JSONParser.scala | 3 +- .../s2graph/core/mysqls/ServiceColumn.scala | 4 +- .../s2graph/core/rest/RequestParser.scala | 10 +- .../apache/s2graph/core/storage/SKeyValue.scala | 4 + .../s2graph/core/storage/StorageSerDe.scala | 2 +- .../hbase/AsynchbaseStorageManagement.scala | 16 +- .../core/storage/rocks/RocksHelper.scala | 2 +- .../storage/rocks/RocksStorageReadable.scala | 54 +- .../vertex/tall/VertexDeserializable.scala | 14 +- .../serde/vertex/tall/VertexSerializable.scala | 20 +- s2jobs/build.sbt | 57 ++ s2jobs/loader.py | 156 ++++ .../hbase/mapreduce/GraphHFileOutputFormat.java | 169 ++++ .../apache/s2graph/s2jobs/S2GraphHelper.scala | 31 + .../s2jobs/loader/GraphFileGenerator.scala | 43 + .../s2jobs/loader/GraphFileOptions.scala | 138 +++ .../s2graph/s2jobs/loader/HFileGenerator.scala | 223 +++++ .../s2jobs/loader/HFileMRGenerator.scala | 161 ++++ .../s2jobs/loader/RawFileGenerator.scala | 51 ++ .../s2jobs/spark/BulkLoadPartitioner.scala | 56 ++ .../s2jobs/spark/FamilyHFileWriteOptions.scala | 35 + .../s2graph/s2jobs/spark/HBaseContext.scala | 850 +++++++++++++++++++ .../s2jobs/spark/HBaseDStreamFunctions.scala | 158 ++++ .../s2jobs/spark/HBaseRDDFunctions.scala | 206 +++++ .../s2graph/s2jobs/spark/JavaHBaseContext.scala | 342 ++++++++ .../s2jobs/spark/KeyFamilyQualifier.scala | 46 + .../s2graph/s2jobs/S2GraphHelperTest.scala | 24 + .../s2jobs/loader/GraphFileGeneratorTest.scala | 256 ++++++ s2rest_play/conf/application.conf | 2 +- 40 files changed, 3625 insertions(+), 152 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-s2graph/blob/ca7adee6/s2core/src/main/scala/org/apache/s2graph/core/mysqls/ServiceColumn.scala ----------------------------------------------------------------------
