Hi Igor, > Is there a faster way to do this?
From my point of view, your current implementation should be fast enough. > When I remove the 'limit 100' it takes too long. The time series has 72 > sensors with 40k datapoints each. For IoTDB, this amount of data is not large. I want to know how much time was spent in executing the query (excluding the time to convert the dataset into a DataFrame). In addition, you can also have a check on the proportion of unsequence data. When there is a lot of unsequence data, the query may be slow. Sequence data and unsequence data are under the folders data/data/sequence and data/data/unsequence respectively. Best, Steve Su ------------------ ???????? ------------------ ??????: "dev" <[email protected]>; ????????: 2021??2??2??(??????) ????8:09 ??????: "dev"<[email protected]>; ????: Re: New here - Python client to pandas Thank you, Xiangdong Here is the python code that I used: # execute sql query statement session_data_set = session.execute_query_statement("select s9,s10,s11,s12,s13,s14,s15,s16,s17,s18,s19,s22,s23,s24,s25,s26,s27,s28,s29,s30,s31,s32,s33,s61,s62,s64,s72,s56,s57 from root.sae.trends.1 order by time desc limit 100") session_data_set.set_fetch_size(1024) results_columns = session_data_set.get_column_names() print(session_data_set.get_column_types()) print(results_columns) results = pd.DataFrame(columns=results_columns) while session_data_set.has_next(): row = session_data_set.next() row_str = row.__str__().split() row_str[0] = datetime.fromtimestamp(row.get_timestamp()/1000).strftime("%Y-%m-%d %H:%M:%S") results = results.append(pd.Series(row_str,index=results_columns),ignore_index=True) print(results) session_data_set.close_operation_handle() When I remove the 'limit 100' it takes too long. The time series has 72 sensors with 40k datapoints each. Is there a faster way to do this? Thank you in advance IoTDB> select count(s1) from root.sae.trends.1 +---------------------------+ |count(root.sae.trends.1.s1)| +---------------------------+ | 40778| +---------------------------+ IoTDB> count timeseries root +-----+ |count| +-----+ | 72| +-----+ Att, Igor Gois Am Di., 2. Feb. 2021 um 02:00 Uhr schrieb Xiangdong Huang < [email protected]>: > By the way, indeed we should consider whether our iotdb-session python API > is efficient enough (we have many array operations and serialization > operations in the Python lib). > > As I am not an expert of Python, I cannot get the conclusion. Also call > contributors. > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > ?????? > ???????? ???????? > > > Xiangdong Huang <[email protected]> ??2021??2??2?????? ????12:56?????? > > > Hi Igor, > > > > > I am sorry if this is not the right place to ask > > > > The mailing list is absolutely the right place:D > > But the problem is our mailing list only supports plain texts. > > If you have to upload screenshots, please create a jira ( > > https://issues.apache.org/jira/projects/IOTDB/issues). > > > > Actually several days ago, my mate @[email protected] > > <[email protected]> also discussed with me about integration IoTDB > > python with Pandas. > > I think it is a great idea, and we can develop it together. > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > ?????? > > ???????? ???????? > > > > > > Igor Gois <[email protected]> ??2021??2??2?????? ????6:23?????? > > > >> Hi everyone, > >> > >> I am sorry if this is not the right place to ask, but I couldn't install > >> QQ or we chat. I am from Brazil. > >> > >> I am using a python client and I would like to know if there is a way to > >> transform a session dataset (session.execute_query_statement) into a > pandas > >> dataframe? > >> > >> I tried with the example code in the while loop. It worked, but it was > >> not so fast. > >> > >> I am in a project to store and analyze wind turbines data and iotdb > seems > >> to be a great fit. > >> > >> Thanks in advance and sorry if this is the wrong place > >> > >> [image: image.png] > >> > >> > >> Igor Gois > >> > >> > >> > >> > >> >
