amygbAI commented on PR #518: URL: https://github.com/apache/incubator-graphar/pull/518#issuecomment-2163388433
alritey folks .. done On Wed, Jun 12, 2024 at 4:39 PM Weibin Zeng ***@***.***> wrote: > please send me an example for spark + csv and i ll refactor it > … <#m_-7895995846438427138_> > On Wed, Jun 12, 2024 at 2:51 PM Semyon *@*.*> wrote: May we add also a > test for graphar spark+json? — Reply to this email directly, view it on > GitHub <#518 (comment) > <https://github.com/apache/incubator-graphar/pull/518#issuecomment-2162529195>>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU > <https://github.com/notifications/unsubscribe-auth/ATIQOSBBSXKDTSQNNEBHTVLZHAHJ3AVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSGUZDSMJZGU> > . You are receiving this because you were mentioned.Message ID: @.*> > > You can refer to the existed test case an just add a test with updated > json test data like: > > - Spark > > https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/maven-projects/spark/graphar/src/test/scala/org/apache/graphar/TestReader.scala#L99-L117 > > json test can be: > > test("read vertex chunks") {// construct the vertex information > val prefix = testData + "/ldbc_sample/json/" > val vertex_yaml = prefix + "Person.vertex.yml" > val vertex_info = VertexInfo.loadVertexInfo(vertex_yaml, spark) > > // construct the vertex reader > val reader = new VertexReader(prefix, vertex_info, spark) > > // test reading the number of vertices > assert(reader.readVerticesNumber() == 903) > val property_group = vertex_info.getPropertyGroup("gender") > > // test reading a single property chunk > val single_chunk_df = reader.readVertexPropertyChunk(property_group, 0) > assert(single_chunk_df.columns.length == 4) > assert(single_chunk_df.count() == 100) > val cond = "gender = 'female'" > var df_pd = single_chunk_df.select("firstName", "gender").filter(cond) > > > - pyspark > > https://github.com/apache/incubator-graphar/blob/ff057cf9bfb3325e2822b31a3e3041e3314c2d97/pyspark/tests/test_reader.py#L29-L67 > > json test can be: > > def test_vertex_reader_with_json(spark): > initialize(spark) > > vertex_info = VertexInfo.load_vertex_info( > GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/") > .joinpath("Person.vertex.yml") > .absolute() > .__str__() > ) > vertex_reader = VertexReader.from_python( > GRAPHAR_TESTS_EXAMPLES.joinpath("/ldbc_sample/json/").absolute().__str__(), > vertex_info, > ) > assert VertexReader.from_scala(vertex_reader.to_scala()) is not None > assert vertex_reader.read_vertices_number() > 0 > assert ( > vertex_reader.read_vertex_property_group( > vertex_info.get_property_group("name") > ).count() > > 0 > ) > assert ( > vertex_reader.read_vertex_property_chunk( > vertex_info.get_property_groups()[0], 0 > ).count() > > 0 > ) > assert ( > vertex_reader.read_all_vertex_property_groups().count() > >= vertex_reader.read_vertex_property_group( > vertex_info.get_property_group("age") > ).count() > ) > assert ( > vertex_reader.read_multiple_vertex_property_groups( > [vertex_info.get_property_group("name")] > ).count() > > 0 > ) > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/incubator-graphar/pull/518#issuecomment-2162737305>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ATIQOSCZKVPDSRXA7AYB56DZHAT7JAVCNFSM6AAAAABJDPBQKGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRSG4ZTOMZQGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
