Re: RFC: using C++ mini cluster implementation in Java and Python tests

t level for faster >> scans and inserts, and making testing simpler and more consistent >> would make things easier to test across C++ and Python. >> >> - Wes >> >> On Mon, Aug 28, 2017 at 2:22 AM, Adar Lieber-Dembo <a...@cloudera.com> >> wrote: &g

Re: [VOTE] Apache Kudu 1.5.0 RC2

-1 I agree with Hao's assessment that KUDU-2131 should be considered a blocker for 1.5. Below is some more color: Before the fix for KUDU-1726, tablet copies fsynced more than necessary but reliably kept their sessions alive by amortizing the cost of the fsync in each downloaded block. With the

RFC: using C++ mini cluster implementation in Java and Python tests

Over the weekend I've been hacking on an idea that's been discussed in the past: replacing our language-specific mini cluster support with the more robust and featureful C++ mini cluster. Besides reducing the maintenance burden for non-C++ language support, this has the added benefit of providing

Re: RFC: using C++ mini cluster implementation in Java and Python tests

The Python patch wasn't as much as work as I thought, so I ended up doing it myself: https://gerrit.cloudera.org/c/8236. On Fri, Oct 6, 2017 at 3:53 PM, Adar Lieber-Dembo <a...@cloudera.com> wrote: > To close the loop on this, I just merged a new action for the Kudu CLI > which

Re: ADD me to JIRA

Do you have an existing account in Apache JIRA? I looked for your first name and your e-mail address but I couldn't find any accounts. If you don't have an account yet, you can create one here: https://issues.apache.org/jira/secure/Signup!default.jspa. Once you have an account created, please

Re: Kudu 1.7.1 Release

Thanks for volunteering, Grant. Here are a couple of other patches to consider. None of them will affect production use cases but improve the robustness of various test or test-running scripts: 3d73405 (build-and-test.sh: unbreak Python 2.6 based builds): needed for build-and-test.sh to pass on

Heads-up: ccache now used when building on el6

I just merged a fix for KUDU-1961, which enables ccache in conjunction with devtoolset-3 when building in RHEL and CentOS 6 environments. I don't expect it to cause any problems, but let me know if you encounter unusual build failures on el6.

Re: Kudu: potential metadata changes

Thanks for starting a discussion on this topic. > There've been a few discussions recently regarding changes to Kudu's > metadata storage. There are a number of areas that could benefit from > improving this layer, and I've been coalescing some of these ideas to lock > down what changes make

Re: Not able to build on Ubuntu

That's usually a sign that download-thirdparty.sh (invoked by build-if-necessary.sh) was interrupted the first time it ran, perhaps via Ctrl-C. Run 'git clean -xdf thirdparty' to clear out your thirdparty build output and try again. If the issue persists, please reply and attach the full output

Re: [VOTE] Apache Kudu 1.7.0 RC1

-1, because commit 038c3eb has messed up deadline-based waiting on macOS. I don't know the full extent of the breakage but Will and Alexey reported seeing abnormally high CPU usage in Kudu servers. I published a fix in http://gerrit.cloudera.org:8080/9695 and, once it is merged to master, I

Re: Flaky pre-commits

> I do think the 'NTP in kudu' could help a bit here, especially if it were > only used as a "backup" in case the kernel is unsynchronized. I'm a little > nervous about the impact on NTP servers, though, in our minicluster based > tests where we might start and stop tens of thousands of times in

Re: Flaky pre-commits

The clock sync errors do seem to have increased over the past few months. If we could just fix those, I think we'd be left with almost entirely "known" flakies. Any ideas as to what's going on? Think it's something that could be addressed with your NTP-client-in-Kudu patch series? On Fri, Mar 23,

Re: [VOTE] Apache Kudu 1.7.0 RC2

+1 - Built in TSAN mode on Ubuntu 16.04 - Ran tests. Most passed except master-test (filed KUDU-2361). Then my computer crashed, taking down the 1.7 build output (which was in /tmp) with it. On Tue, Mar 20, 2018 at 10:09 AM, William Berkeley wrote: > +1 > > - Built in

Re: [VOTE] Apache Kudu 1.8.0-RC2

+1 Built and ran DEBUG C++ tests in slow mode on CentOS 6.6. All passed. Built and ran Java tests using DEBUG Kudu binaries on CentOS 6.6. All passed. Reviewed the release notes. On Mon, Oct 22, 2018 at 5:17 PM Dan Burkert wrote: > > +1 > > Built and ran C++ tests on MacOS. Had three failures

Re: Kudu 1.8.0 Release Notes Draft

Thanks for doing this, Attila. I've added release notes for my patches, Todd's patches (since he's currently away), and for all Kudu contributors who aren't committers. On Wed, Oct 3, 2018 at 2:07 PM Attila Bukor wrote: > > Hi Devs, > > I've created a Google doc to collect the release notes: >

Re: Adding more timestamp types to on-disk storage formats

Hi Zoltan, I'm not feeling brave enough to continue cc'ing a bunch of other Apache projects just yet, but I did leave a comment in your gdoc describing what Kudu currently supports and what we could offer in the future. It wasn't clear if you wanted to carry out this discussion over gdoc

Re: Kudu 1.9.0 Release

+1 Thanks for volunteering to RM, Andrew. On Tue, Jan 22, 2019 at 9:35 PM Andrew Wong wrote: > > Hello Kudu developers! > > It's been just around three months since we released 1.8.0. In that time, > we've accrued some important improvements and bug fixes (around 201 commits > since Kudu

Re: [NOTICE] Mandatory migration of git repositories to gitbox.apache.org

Do we need to update any configuration in gerrit.cloudera.org or jenkins.kudu.apache.org? On Thu, Jan 3, 2019 at 6:18 AM Attila Bukor wrote: > Hi, > > I submitted 2 patches to Gerrit to change our tooling to use the gitbox > URLs: > > - https://gerrit.cloudera.org/c/12150/ > -

Heads-up: dropping JDK 7 support in the next Kudu release

I just merged a patch to master that drops JDK 7 support from Kudu's Java artifacts (i.e. the Java client, Spark bindings, etc.). The oldest supported JDK version is now JDK 8. JDK 7 support has been deprecated since Kudu 1.5.0 (about a year and a half ago) so there's been ample notice that this

Java tests now tracked by flaky test dashboard

I merged a series of patches that will cause Java test failures to be reported to the flaky test dashboard (http://dist-test.cloudera.org:8080). Over the next week or so we should start to see failures percolate in and learn more about which tests are flaky. Of note: we used to retry every Java

Kudu backport policy

A recent Slack discussion revealed that we may not have consensus on how our project handles backports to older branches. This e-mail is an attempt to build that consensus; please weigh in if you have an opinion. Perhaps it's best to start from first principles: when release x.y is the latest

Re: [VOTE] Apache Kudu 1.9.0-RC2

+1 I ran the C++ tests (DEBUG build) in slow mode on Ubuntu 18, CentOS 6.6, and CentOS 7.3. I saw a couple failures but I'm pretty sure they're just flakes; filed KUDU-2718 and KUDU-2719 for them. On Wed, Feb 27, 2019 at 1:51 AM Andrew Wong wrote: > > Hello Kudu devs! > > As suggested on the

Re: [VOTE] Apache Kudu 1.9.0-RC1

+1 I ran all C++ tests with KUDU_ALLOW_SLOW_TESTS=1 in DEBUG mode on Ubuntu 18.04 and CentOS 6.6. All passed except for a known Ubuntu 18 failure (a variant of KUDU-2641). On Thu, Feb 21, 2019 at 5:54 PM helifu wrote: > > +1 > * All C++ tests passed in debug/release/tsan mode on Debian8.9,

Re: Simulating failures in krpc

> > I'm an Impala dev working on replacing Thrift with krpc. One issue that > > recently came up is that we would like to have a simple way of simulating > > different types of failures of rpcs for testing purposes, and I was > > wondering if krpc already has anything like this built in, or if

Re: java unit tests failing when starting MiniKuduCluster

build), with and without my patch. > > ________ > From: Adar Lieber-Dembo > Sent: 14 April 2019 19:59:45 > To: dev@kudu.apache.org > Subject: Re: java unit tests failing when starting MiniKuduCluster > > It might be related to your local changes. This caught

Re: [ANNOUNCE] Welcoming Yingchun Lai as a Kudu committer and PMC member

Congrats, Yingchun! Thank you for all of your hard work on Kudu. On Wed, Jun 5, 2019 at 11:25 AM Todd Lipcon wrote: > > Hi Kudu community, > > I'm happy to announce that the Kudu PMC has voted to add Yingchun Lai as a > new committer and PMC member. > > Yingchun has been contributing to Kudu for

Re: java unit tests failing when starting MiniKuduCluster

It might be related to your local changes. This caught my eye: extra_master_flags: "--rpc_max_message_size=3860733440" extra_tserver_flags: "--rpc_max_message_size=3860733440" Why did you have to add this configuration? What happens if you remove it? On a related note, do the tests still

Flaky test dashboard results are going to change

Today I finished merging a patch series that switches on dist-test when running tests that power the flaky test dashboard (previously they were run on a CentOS 6.6 VM). This wasn't done because we need these tests to finish faster; it's because an increasing number of tests are flaky in one

Re: Dimension table delete and recreate

(+user, -dev, as this is more appropriate for the users list) The Kudu master currently keeps a record of all tables and partitions, including those that have been deleted. With a high enough rate of table deletion it's theoretically possible for that to consume a lot of disk space or memory. In

[ANNOUNCE] Welcoming Lifu He, Yao Xu, and Yao Zhang as Kudu committers and PMC members

Hi Kudu community, I'm happy to announce that the Kudu PMC has voted to add Lifu He, Yao Xu, and Yao Zhang as new committers and PMC members. Lifu has worked on a variety of patches, from dead container deletion in the block manager, to metric aggregation in the master, to several performance

Re: Kudu 1.11.0 Release

+1 On Tue, Oct 1, 2019 at 11:44 AM Alexey Serbin wrote: > > Hello Kudu developers! > > It's been a bit more than three months since we released 1.10.0. In that > time, > we've accrued some important improvements and bug fixes (around 189 commits > since Kudu 1.10.0). By our usual three-month

Re: Licensing issue in Kudu 1.10.0 and 1.11.0

My two cents: - The presence of 1.11.0 on the download page means that 1.11.0 has been de facto released, announcement or no announcement. The announcement doesn't add any additional hurt, so I think we should move forward with it. - Separately, let's also announce the licensing issue and say that

JIRA integration with gitbox

Hi devs, The Apache Impala project has this neat integration set up between gitbox and JIRA where if someone pushes a new commit to gitbox and that commit has a JIRA associated with it, the push generates a JIRA comment describing the commit that was just pushed. You can see an example of this

Re: JIRA integration with gitbox

: > > > >> > > > >> SGTM. My first thought was that the issues mailing list might get more > > > >> traffic, but I'm ok with that, since contributors today often try to > > > post > > > >> an equivalent comment anyways.

Re: Licensing issue in Kudu 1.10.0 and 1.11.0

UDU-2990 and the rest of notes for > 1.11.0 and 1.10.0 correspondingly? > > > Thanks, > > Alexey > > On Thu, Nov 7, 2019 at 5:07 AM Adar Lieber-Dembo > wrote: > > > KUDU-2990 has been fixed on master, so 1.12.0 will be conformant when > > it is released in s

Re: Flume Integration Removal

removing it from Kudu. > >> > >> > >> I am okay with that. I will post a patch to the project and share it > >> here. > >> > >> On Wed, Sep 18, 2019 at 2:05 PM Adar Lieber-Dembo > >> wrote: > >> > >>>

Re: [VOTE] Apache Kudu 1.11.1-RC1

+1 On Ubuntu 18.04 I built RELEASE with NO_TESTS=1 and verified that there were no numa or memkind dependencies or symbols in any binaries. I also grepped for numa and memkind across the codebase and didn't see any unexpected dependencies. I did see memkind erroneously listed in

Re: [VOTE] Apache Kudu 1.11.1-RC2

+1 I ran DEBUG tests in slow mode on Ubuntu 18.04 and CentOS 6.6. All passed. On Sat, Nov 16, 2019 at 1:38 PM Attila Bukor wrote: > > +1 > > - Verified checksum, the signature and that the archive was created from the > tag > - Successfully built Kudu in release mode on CentOS 7.3 and macOS

Re: [VOTE] Apache Kudu 1.10.1-RC2

+1 On Ubuntu 18.04 I built RELEASE with NO_TESTS=1 and verified that there were no numa or memkind dependencies or symbols in any binaries. I also grepped for numa and memkind across the codebase and didn't see any unexpected dependencies. Also on Ubuntu 18.04 I built DEBUG and verified that all

Re: Licensing issue in Kudu 1.10.0 and 1.11.0

u.apache.org/docs/known_issues.html#_other_known_issues > > > > > > Best regards, > > > > Alexey > > > > On Fri, Nov 1, 2019 at 4:20 PM Adar Lieber-Dembo > > > > wrote: > > > > > My two cents: > > > - The presence of 1

Re: Time source for Kudu tests

o me. > > > > I don't feel particularly strongly for this approach vs the one laid out by > > Alexey, but thought I'd bring it up since in general it seems like external > > mini cluster tests are meant to simulate as real of an environment as > > possible (modulo a

Re: 回复: [VOTE] Apache Kudu 1.11.0-RC3

+1 I ran all tests in slow DEBUG mode on CentOS 6.6 and Ubuntu 18.04. All of them passed. I'm surprised that live row count still has issues given the late breaking fixes that Yingchun Lai merged into 1.11.0. His last fix was expressly intended to address the problem where a table with a mix of

Re: [VOTE] Apache Kudu 1.11.0-RC1

-1 I built and ran all DEBUG tests in slow mode on Ubuntu 18.04 and CentOS 6.6. Notable failures included: - memory_gc-itest: looks like it's broken for most DEBUG builds, even after Yingchun's deflake patch (http://dist-test.cloudera.org:8080/test_drilldown?test_name=memory_gc-itest). -

Re: [VOTE] Apache Kudu 1.11.0-RC2

+0 Built on Ubuntu 18.04 and CentOS 6.6. Ran DEBUG tests in slow mode. No failures. I do wish the release incorporated the fix for KUDU-2980, but that hasn't even been published to CR yet. Plus it won't break backwards compatibility if we fix it later. On the other hand,

Re: Time source for Kudu tests

Thanks for the detailed summary. I agree with the goal of using the same time source across all tests if possible. Originally, I was hoping this would be 'builtin', but that'd require deploying MiniChronyd too, both to avoid a dependency on Internet connectivity and to ensure that clock

[ANNOUNCE] Welcoming Yifan Zhang as Kudu committer and PMC member

Hi Kudu community, I'm happy to announce that the Kudu PMC has voted to add Yifan Zhang as a new committer and PMC member. Yifan's contributions have included: - Filtering tables and tablets in ksck. - Tooling to remove or alter columns. - Adding a mean gauge to the metrics subsystem. - The

Re: Hive 3 and Apache Ranger

nd code. > >- This can be done after the next release. > > - Though the tests are removed and we use Hive 3 we still technically > > work with Hive 2 and Sentry. We are no longer testing/validating > > it though. > > > > Please let me know if you

Heads-up: change in include-what-you-use behavior

I just merged some changes that modify include-what-you-use (IWYU) to run against the libc++ in thirdparty rather than against the system's libstdc++. Coupled with some other changes, I expect that running IWYU on any Linux system should yield the exact same set of recommendations as when IWYU is

Re: Switching time source to the built-in NTP client

I share the two concerns you highlighted, Alexey. 1. This would be a backwards incompatible change. 2. The default NTP server list could be unreachable, or could be a poorer choice than whatever the cluster is currently using. Grant's suggestion could mitigate that somewhat, but it's sort of weird