Re: Questions about Hive Integration

[田原]

Hi,

1. Hive 3.x needs hadoop 3.x, but our project is based on the hadoop 2.x, and 
these two are incompatible.

2. Now, we don't support insert operation in hive. Only query operation is 
supported.

3. If there are more than one tsfile in one folder, all the tsfiles in that 
folder will be pre-read by InputFormat, and then filter them by the table 
name(i.e. device_id) and the field (i.e. sensor_id). So the result set returned 
to user will only contains data they want.


> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2019-10-12 14:38:36 (星期六)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Questions about Hive Integration
> 
> Hi,
> 
> Today I meets some users from Inspur (a famous Chinese company), and they
> address some requirements about IoTDB with Hive.
> 
> 1. First, they use hadoop 3.x.
> 
> 2. Can we use Hive to write data back to HDFS with TsFile format?
> e.g., HiveQL: create table . LOCATION HDFS_FILE
> 
> 3. If Hive user uses Load command from a IoTDB's data folder, what happens?
> E.g., HiveQL: load data inpath 'iotdb/data/root.sg1/' into table
> tablename...
> Then, which Table files will be observed by Hive? If more TsFiles in the
> folder are generated by IoTDB, can Hive observes them automatically?
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院

Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release

[Xiangdong Huang]

Hi Justin,

In my understanding (I am not sure whether my understanding is correct),
uploading files to the maven Nexus Staging repository is optional...

The only thing we must to do before starting a voting is uploading the
source-release.zip with its asc and sha512 files  (and the
binary-distribtuion.zip if wanted) to https://dist.apache.org/repos/dist/dev
.

There are two ways to generate the source-release.zip:

(1) do it manually, and generate asc and sha512 files manually.

(2) using mvn release:prepare and release:perform command to do that
automatically (maven will organize files according to the description of
distribution.xml, copying jars into the lib folder and copying scripts to
the correct location according to assembly folder and server.xml and
client.xml)
The side-effect is that, mvn release command will upload the jars into the
maven nexus staging repository... (fortunately, users can not observe these
files before I manually click  "release" button, so that maybe we can bear
the side-effect)

I think doing it automatically is good. If there is other ways to do that
automatically, I will try it.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Justin Mclean  于2019年10月13日周日 上午9:04写道：

> Hi,
>
> I’m still not sure why you are preparing the maven artefacts before the
> vote is finished. I guess it good to check that everyone is OK but they can
> be prepared after as maven is not an official release area.
>
> Thanks,
> Justin

Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release

[Justin Mclean]

Hi,

I’m still not sure why you are preparing the maven artefacts before the vote is 
finished. I guess it good to check that everyone is OK but they can be prepared 
after as maven is not an official release area.

Thanks,
Justin

Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release

[Xiangdong Huang]

Hi,

> I am not quite sure, but I guess when preparing the release the steps
haven't been executed as written in the RM guide.

When release RC1, I ran:
1. mvn release:prepare -P apache-release  -DautoVersionSubmodules=true
2. mvn release:perform -DreleaseProfiles=apache-release

Because RC1 has some other issues, I had to prepare RC2.
Then I found that I can not run `mvn release:rollback` (because all
pom.xml.releaseBackup had been cleaned automatically).
So I checked rel/0.8's latest codes from the repo (and the version was
changed to 0.8.2-SNAPSHOT by the `mvn release:perform` command).
I had to change the version number back to 0.8.1-SNAPSHOT and then run `mvn
release:prepare` and `mvn release:perform` once again.

That is all what I did using maven... Is there something incorrect?

One more thing, I checked the RC1 source-release.zip file[1], the
maven-wrapper.jar also exists...

[1]
https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.8.1/rc1/apache-iotdb-0.8.1-incubating-source-release.zip

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Christofer Dutz  于2019年10月12日周六 下午7:30写道：

> Yeah ... I wanted to say, the .gitignore helps preventing to accidentally
> check it in, but it doesn't automatically exclude it from the source
> assembly.
>
> Guess I never before used the maven wrapper for building a release (Always
> have maven installed) ...
>
> But thinking about it ... this file actually should never have been there
> in the first place ...
> The release plugin does the preparing of a release (change the version to
> a non-SNAPSHOT version) and then tags this version and then changes to the
> next development version and checks both in.
> The actual release is then built during the release:perform step by
> checking out this tag from git into an empty "target/checkout" directory
> and spawning a new maven build in that directory (without the maven
> wrapper).
> So this jar shouldn't have been there, as it is only downloaded when
> running maven manually using the mvnw script, which the release:perform
> goal of the release plugin doesn't do.
>
> I am not quite sure, but I guess when preparing the release the steps
> haven't been executed as written in the RM guide.
> Using it, the file shouldn't have been there and there shouldn't be the
> need to adjust the assembly descriptor to exclude that file.
>
> Chris
>
>
>
>
>
> Am 12.10.19, 05:56 schrieb "Willem Jiang" :
>
> en, Justin, you are right. We need to update the distribution assembly
> file at the same time.
> I just update the JIRA issue about this.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Sat, Oct 12, 2019 at 10:22 AM Justin Mclean <
> jus...@classsoftware.com> wrote:
> >
> > HI,
> >
> > > For the jar in the source release, we could address it by adding
> below
> > > line into .gitignore file.
> > > #skip the maven-wrapper.jar
> > > .mvn/wrapper/maven-wrapper.jar
> >
> > How would that fix it? The release needs to be be made on on the RM
> machine not from guthub and GitHub doesn’t currently have that jar. [1]
> >
> > Thanks,
> > Justin
> >
> > 1.
> https://github.com/apache/incubator-iotdb/tree/master/.mvn/wrapper
>
>
>

Re: More programming language APIs generated by Thrift

[Xiangdong Huang]

Hi Chris,

Of course I want to compile the generated codes... Please give us help. :D

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Christofer Dutz  于2019年10月12日周六 下午5:06写道：

> Well it generates the api, but doesn't build it as a module, correct?
>
> Was not referring to if you wanted to also compile the generated stuff.
>
> Chris
>
> Holen Sie sich Outlook für Android
>
> 
> From: Xiangdong Huang 
> Sent: Saturday, October 12, 2019 9:07:47 AM
> To: dev@iotdb.apache.org 
> Subject: Re: More programming language APIs generated by Thrift
>
> Hi Chris,
>
> I have used maven plugin to generate these language apis in [1]. Are there
> some other suggestions?
>
> [1] https://github.com/apache/incubator-iotdb/pull/445
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> Christofer Dutz  于2019年10月11日周五 下午3:59写道：
>
> > Hi all,
> >
> > having done all the Thrift compilation stuff for PLC4X I guess I might be
> > able to help here.
> >
> > Chris
> >
> > Am 11.10.19, 09:17 schrieb "Xiangdong Huang" :
> >
> > Hi,
> >
> > I notice there are some PRs [2][3] for generating CPP, python apis by
> > using
> > thrift.
> >
> > Actually it can be compiled automatically by Maven.
> >
> > Chris introduced a maven-thrift-plugin several months ago. And, just
> > modifying the configuration can archive that..
> >
> > Using PR [1], we can generate cpp, python, and go apis under
> > server-rpc/target/generated-sources-{language}.
> >
> > For other PRs, maybe you just need to write examples about how to use
> > these
> > api...  (and some docs?)
> >
> >
> > [1] https://github.com/apache/incubator-iotdb/pull/445
> > [2] https://github.com/apache/incubator-iotdb/pull/444
> > [3] https://github.com/apache/incubator-iotdb/pull/415
> >
> > Best,
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> >
> >
> >
>

Re: [discussion] Release of IoTDB 0.8.1 RC2

[Kevin A. McGrail]

Thanks for the clarification!

On Fri, Oct 11, 2019, 22:36 Jialin Qiao  wrote:

> Hi Kevin,
>
> > Question: The release notes don't mention the speed increase.  I thought
> there was a substantial speed increase with 0.8.1.  Did I misunderstand
> that?
>
> The 0.8.1 is just a bug-fix version of 0.8.0, so there isn't any speed
> increasing. Indeed we have a substantial speed increase over 0.8.0, which
> will be contained in the 0.9.0. Looking forward to the release of 0.9.0 :)
>
> > Question: Should the client script be in an sbin directory?  I would
> think just bin would be better.  Not a release blocker.
>
> +1, we could change it later. Like Spark did, sbin includes the server
> script and bin includes client or tools script.
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -原始邮件-
> > 发件人: "Kevin A. McGrail" 
> > 发送时间: 2019-10-11 22:09:28 (星期五)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: Re: [discussion] Release of IoTDB 0.8.1 RC2
> >
> > I vote +1 having followed this list:
> >
> > downloaded pre-release/iotdb-0.8.1 rc2 binary release from
> https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.8.1/rc2/apache-iotdb-0.8.1-incubating-bin.zip
> >
> > confirmed sha512 matches
> >
> 604a9ba330237766f8f3488a13cde63b573ad005f29a3f9748535a5ad65324b5fc75c25d44f805b44919f2b1405fe4889905f0e31948cf13beade48644d08263
> apache-iotdb-0.8.1-incubating-bin.zip
> >
> >
> > I Looked at the NOTICE, LICENSE and DISCLAIMER
> > I loooked at the README.md and the RELEASE_NOTES.md
> >
> > Question: The release notes don't mention the speed increase.  I thought
> there was a substantial speed increase with 0.8.1.  Did I misunderstand
> that?
> >
> > ran start iotdb server
> > start client
> >
> > Question: Should the client script be in an sbin directory?  I would
> think just bin would be better.  Not a release blocker.
> >
> > Executed the following successfully.
> > SET STORAGE GROUP TO root.ln.wf01.wt01;
> > CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN,
> ENCODING=PLAIN
> > CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=FLOAT,
> ENCODING=PLAIN
> > CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32,
> ENCODING=PLAIN
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(1, 1.1, false, 11)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(2, 2.2, true, 22)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(3, 3.3, false, 33 )
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(4, 4.4, false, 44)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(5, 5.5, false, 55)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(6, 1.1, false, 11)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(7, 2.2, true, 22)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(8, 3.3, false, 33 )
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(9, 4.4, false, 44)
> > INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
> values(10, 5.5, false, 55)
> > Select * from root;
> >
> > I did not do the source release as my system is not working correctly
> with mvn but based on others feedback, I feel confident in the binary
> release test.
> >
> > Regards,
> > KAM
> >
> > On 10/10/2019 9:48 PM, Xiangdong Huang wrote:
> > > Hi kevin,
> > >
> > > Ah, as after the voting in this mailing list, we still need to vote on
> > > general@ to get at least 3 +1 from IPMCs.
> > > But I noticed that when Julian released 0.8.0 and got more than 3 +1
> from
> > > IPMCs in dev@iotdb.apache.org mail list,
> > > he started a new lazy vote in the general@, which I thought it is
> because
> > > there were 3 IPMCs voting it already.
> > > I think it is convenient..
> > >
> > > Best,
> > > ---
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > Kevin A. McGrail  于2019年10月11日周五 上午9:01写道：
> > >
> > >> Are we needed for a release vote?  I usually like to make sure you
> guys
> > >> are doing things correctly and try to leave you to your own devices...
> > >>
> > >> On 10/10/2019 8:49 PM, Xiangdong Huang wrote:
> > >>> Hi,
> > >>>
> > >>> There are 7 PPMCs voting for the release now. How about IPMC and
> mentors?
> > >>>
> > >>> Best,
> > >>> ---
> > >>> Xiangdong Huang
> > >>> School of Software, Tsinghua University
> > >>>
> > >>>  黄向东
> > >>> 清华大学 软件学院
> > >>>
> > >>>
> > >>> Xiangdong Huang  于2019年10月9日周三 下午2:21写道：
> > >>>
> >  Hi,
> > 
> >  There are 3 PPMCs voting for the release now. How about other PPMCs?
> > 
> >  Best,
> >  ---
> >  Xiangdong Huang
> >  School of Software, Tsi

Re: [VOTE] Apache IoTDB 0.8.1 (incubating) RC2 release

[Christofer Dutz]

Yeah ... I wanted to say, the .gitignore helps preventing to accidentally check 
it in, but it doesn't automatically exclude it from the source assembly.

Guess I never before used the maven wrapper for building a release (Always have 
maven installed) ... 

But thinking about it ... this file actually should never have been there in 
the first place ...
The release plugin does the preparing of a release (change the version to a 
non-SNAPSHOT version) and then tags this version and then changes to the next 
development version and checks both in.
The actual release is then built during the release:perform step by checking 
out this tag from git into an empty "target/checkout" directory and spawning a 
new maven build in that directory (without the maven wrapper). 
So this jar shouldn't have been there, as it is only downloaded when running 
maven manually using the mvnw script, which the release:perform goal of the 
release plugin doesn't do.

I am not quite sure, but I guess when preparing the release the steps haven't 
been executed as written in the RM guide. 
Using it, the file shouldn't have been there and there shouldn't be the need to 
adjust the assembly descriptor to exclude that file.

Chris





Am 12.10.19, 05:56 schrieb "Willem Jiang" :

en, Justin, you are right. We need to update the distribution assembly
file at the same time.
I just update the JIRA issue about this.

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Sat, Oct 12, 2019 at 10:22 AM Justin Mclean  
wrote:
>
> HI,
>
> > For the jar in the source release, we could address it by adding below
> > line into .gitignore file.
> > #skip the maven-wrapper.jar
> > .mvn/wrapper/maven-wrapper.jar
>
> How would that fix it? The release needs to be be made on on the RM 
machine not from guthub and GitHub doesn’t currently have that jar. [1]
>
> Thanks,
> Justin
>
> 1. https://github.com/apache/incubator-iotdb/tree/master/.mvn/wrapper

Re: More programming language APIs generated by Thrift

[Christofer Dutz]

Well it generates the api, but doesn't build it as a module, correct?

Was not referring to if you wanted to also compile the generated stuff.

Chris

Holen Sie sich Outlook für Android


From: Xiangdong Huang 
Sent: Saturday, October 12, 2019 9:07:47 AM
To: dev@iotdb.apache.org 
Subject: Re: More programming language APIs generated by Thrift

Hi Chris,

I have used maven plugin to generate these language apis in [1]. Are there
some other suggestions?

[1] https://github.com/apache/incubator-iotdb/pull/445

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Christofer Dutz  于2019年10月11日周五 下午3:59写道：

> Hi all,
>
> having done all the Thrift compilation stuff for PLC4X I guess I might be
> able to help here.
>
> Chris
>
> Am 11.10.19, 09:17 schrieb "Xiangdong Huang" :
>
> Hi,
>
> I notice there are some PRs [2][3] for generating CPP, python apis by
> using
> thrift.
>
> Actually it can be compiled automatically by Maven.
>
> Chris introduced a maven-thrift-plugin several months ago. And, just
> modifying the configuration can archive that..
>
> Using PR [1], we can generate cpp, python, and go apis under
> server-rpc/target/generated-sources-{language}.
>
> For other PRs, maybe you just need to write examples about how to use
> these
> api...  (and some docs?)
>
>
> [1] https://github.com/apache/incubator-iotdb/pull/445
> [2] https://github.com/apache/incubator-iotdb/pull/444
> [3] https://github.com/apache/incubator-iotdb/pull/415
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
>

Start working on supporting dot symbol for measurement name

[康愈圆]

Hi, all


I'm now working on [IOTDB-38], allowing dot symbol for measurement name.
Url: 
https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-38?filter=allissues


Best,
_
Yuyuan KANG

Re: More programming language APIs generated by Thrift

[Xiangdong Huang]

Hi Chris,

I have used maven plugin to generate these language apis in [1]. Are there
some other suggestions?

[1] https://github.com/apache/incubator-iotdb/pull/445

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Christofer Dutz  于2019年10月11日周五 下午3:59写道：

> Hi all,
>
> having done all the Thrift compilation stuff for PLC4X I guess I might be
> able to help here.
>
> Chris
>
> Am 11.10.19, 09:17 schrieb "Xiangdong Huang" :
>
> Hi,
>
> I notice there are some PRs [2][3] for generating CPP, python apis by
> using
> thrift.
>
> Actually it can be compiled automatically by Maven.
>
> Chris introduced a maven-thrift-plugin several months ago. And, just
> modifying the configuration can archive that..
>
> Using PR [1], we can generate cpp, python, and go apis under
> server-rpc/target/generated-sources-{language}.
>
> For other PRs, maybe you just need to write examples about how to use
> these
> api...  (and some docs?)
>
>
> [1] https://github.com/apache/incubator-iotdb/pull/445
> [2] https://github.com/apache/incubator-iotdb/pull/444
> [3] https://github.com/apache/incubator-iotdb/pull/415
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
>

[jira] [Created] (IOTDB-251) Improve query data structure in RPC

[Jialin Qiao (Jira)]

Jialin Qiao created IOTDB-251:
-

 Summary: Improve query data structure in RPC
 Key: IOTDB-251
 URL: https://issues.apache.org/jira/browse/IOTDB-251
 Project: Apache IoTDB
  Issue Type: Improvement
Reporter: Jialin Qiao


Similar to the RowBatch in the write interface, we could use primitive data 
arrays in ResultSet in the RPC query, instead of constructing record each line.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)