Hi Weibin, Is this issue resolved? Also, is it the same for Edges too? For example if I have an edge chunk size of 1 Million and there are 990K edges, to add one more edge to the same edge collection with builder, do I have read out all the edges and add the new edge and write back? Alternately, is there a workaround to add a new chunk while creating new vertices or edges even if the last chunk is not full? For example, if I choose vertex chunk size and edge chunk size as 1000, say there are 5500 vertices and 23400 edges, can I add a new vertex in chunk 7 starting with id 6000 and new edge in chunk 25 starting at 24000? If so, does vertex_count and edge_count still work?
Thanks, Muni ________________________________ From: Muniraju Pulipalyam <munir...@hotmail.com> Sent: Wednesday, August 28, 2024 1:56 PM To: dev@graphar.apache.org <dev@graphar.apache.org> Subject: Re: Add a new vertex to an existing graph of vertices and edges. OK. What's the API to read the last chunk? Is it with VerticesCollection? VerticesCollection seems to be very slow. I looked at mid_level _reader sample, but that seems to be about one property. ________________________________ From: Weibin Zeng <wei...@apache.org> Sent: Wednesday, August 28, 2024 12:30:30 PM To: dev@graphar.apache.org <dev@graphar.apache.org> Subject: Re: Add a new vertex to an existing graph of vertices and edges. Hi，Muni, Currently graphar doesn't support appending vertices to a existed chunk (which is a existed parquet or orc file) but new a chunk. So the best way is read the chunk you want to append and append to the chunk in memory, then write it out. I think this is the case that we need to address in the library, which no need user to take care the process. I will open an issue for this feature. Thanks, Weibin Zeng On 2024/08/28 03:05:54 Muniraju Pulipalyam wrote: > Hello all/ Weibin, > > Can someone please help me with this query below? Say, I have a chunk size > of 1024. If I have 5000 vertices in my graph, 4 chunks will be of 1024 > vertices length and 5th chunk will be 904 vertices. When I try to write > 5001st item, if I pass 5001 as the start vertex index to VerticeBuilder, it > is overriding 904 elements with 1 element I am adding. Should I read the > vertices of last chunk, add to the builder and then append back? > > Thanks, > Muni > > ________________________________ > From: Muniraju Pulipalyam > Sent: Monday, August 26, 2024 8:49 PM > To: dev@graphar.apache.org <dev@graphar.apache.org> > Subject: Re: Add a new vertex to an existing graph of vertices and edges. > > Thanks Weibin. This seems to work only if vertex chunk size is 1, if I want > to add 1 vertex to an existing graph with vertices when I have strong > validation. When chunk size is 1 it creates new file and everything works. > But if chunk size is not 1 , if I enable strong validation, add vertex fails > with "the start vertex index <my index number> is not aligned with chunk > size" in verticesbuulder.ccline 44. If I don't enable strong validation, then > last chunk file is overwritten with 1 row data, when I call dump on > vertexBuilder. I tried with 10 row vertices ,chunk size 5. 11th row succeeds > with new chunk created. But, if I change chunk size to 4, then I get above > validation error > > Sent from Outlook for > Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C02%7C%7C22d457b4dd024c368a5808dcc72f1c83%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638604252363995851%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=L9yQYw0Q6boAT5h6yIMnRVPBjLlDcdDIjIoOwj2nDf8%3D&reserved=0<https://aka.ms/AAb9ysg>> > ________________________________ > From: Weibin Zeng <wei...@apache.org> > Sent: Monday, August 26, 2024 8:46:40 AM > To: dev@graphar.apache.org <dev@graphar.apache.org> > Subject: Re: Add a new vertex to an existing graph of vertices and edges. > > You need to specify the start vertex index when you construct the > `VerticesBuilder`[1] if you want append vertices to existed graph. In your > senario, the start vertex index should be 5001. > > [1] > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgraphar.apache.org%2Fdocs%2Fcpp%2Fclassgraphar_1_1builder_1_1_vertices_builder.html%23a8215ee434947314027aae3a7e10dac5b&data=05%7C02%7C%7C22d457b4dd024c368a5808dcc72f1c83%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638604252364002978%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6zM%2Bp7Ff%2Bhq08Byrk%2BPYQMDIF2lL1VqIY0ENvUHRrlM%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgraphar.apache.org%2Fdocs%2Fcpp%2Fclassgraphar_1_1builder_1_1_vertices_builder.html%23a8215ee434947314027aae3a7e10dac5b&data=05%7C02%7C%7C22d457b4dd024c368a5808dcc72f1c83%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638604252364005896%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=4VUgHmbYXqI3jGE8xLNEQKbmYc5Zm40s0z%2BNm8nhii4%3D&reserved=0><https://graphar.apache.org/docs/cpp/classgraphar_1_1builder_1_1_vertices_builder.html#a8215ee434947314027aae3a7e10dac5b> > > Best, > Weibin Zeng > > On 2024/08/25 06:51:45 Muniraju Pulipalyam wrote: > > I have an existing graph with 5000 vertices and 80000 edges. The vertices > > are in 4 chunks. When I try to use verticesBuilder and create 5001st vertex > > to add to existing collection, it is overriding existing vertex chunks. I > > also tried WriteTable, but can't find a way to append. I saw samples on > > mid-level and high-level writers and other samples where new files are > > created in /tmp folder. I want to append a vertex to an existing vertex > > collection. What API can I use to append a vertex to vertex collection and > > write back to file? > > > > Thanks. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org > For additional commands, e-mail: dev-h...@graphar.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org For additional commands, e-mail: dev-h...@graphar.apache.org
