acezen commented on PR #518: URL: https://github.com/apache/incubator-graphar/pull/518#issuecomment-2162737305
> please send me an example for spark + csv and i ll refactor it > […](#) > On Wed, Jun 12, 2024 at 2:51 PM Semyon ***@***.***> wrote: May we add also a test for graphar spark+json? — Reply to this email directly, view it on GitHub <[#518 (comment)](https://github.com/apache/incubator-graphar/pull/518#issuecomment-2162529195)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***> You can refer to the existed test case an just add a test with updated json test data like: - Spark https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala#L99-L117 json test can be: ```scala test("read vertex chunks") { // construct the vertex information val prefix = testData + "/ldbc_sample/json/" val vertex_yaml = prefix + "Person.vertex.yml" val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) // construct the vertex reader val reader = new VertexReader(prefix, vertex_info, spark) // test reading the number of vertices assert(reader.readVerticesNumber() == 903) val property_group = vertex_info.getPropertyGroup("gender") // test reading a single property chunk val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) assert(single_chunk_df.columns.length == 4) assert(single_chunk_df.count() == 100) val cond = "gender = 'female'" var df_pd = single_chunk_df.select("firstName", "gender").filter(cond) ``` - pyspark https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/pyspark/tests/test_reader.py#L29-L67 json test can be: ```python def test_vertex_reader_with_json(spark): initialize(spark) vertex_info = VertexInfo.load_vertex_info( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/") .joinpath("Person.vertex.yml") .absolute() .__str__() ) vertex_reader = VertexReader.from_python( GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().__str__(), vertex_info, ) assert VertexReader.from_scala(vertex_reader.to_scala()) is not None assert vertex_reader.read_vertices_number() > 0 assert ( vertex_reader.read_vertex_property_group( vertex_info.get_property_group("name") ).count() > 0 ) assert ( vertex_reader.read_vertex_property_chunk( vertex_info.get_property_groups()[0], 0 ).count() > 0 ) assert ( vertex_reader.read_all_vertex_property_groups().count() >= vertex_reader.read_vertex_property_group( vertex_info.get_property_group("age") ).count() ) assert ( vertex_reader.read_multiple_vertex_property_groups( [vertex_info.get_property_group("name")] ).count() > 0 ) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
