I'm still working on this, but I struggle a lot with the Jenkinsfile, and some Yetus bugs also make testing hard.
On Thu, Sep 19, 2024 at 9:25 AM Istvan Toth <st...@cloudera.com> wrote: > Master is still going to support 3.3.5, so master still needs to use > reflection as well. > > The point of these changes is allowing Hbase to default to a newer Hadoop > version, without dropping support for older releases. > > Dropping support for 3.3.5 would be a different discussion, and I > personally feel that it would be too early. > > Istvan > > On Wed, Sep 18, 2024 at 7:46 PM Wei-Chiu Chuang > <weic...@cloudera.com.invalid> wrote: > >> So....I was wondering now that we'll be using Hadoop 3.4.0, if it's okay >> to >> port HBASE-27769 <https://issues.apache.org/jira/browse/HBASE-27769> to >> the >> master branch. >> This will allow Ozone to be used by HBase. We are preparing Apache Ozone >> 2.0 release and having a usable Apache HBase to work with is important. It >> is working now with Cloudera's HBase work but I'd like to open up this >> opportunity to the community as well. >> >> We can start with master, and then I can find a solution (something that >> involves the use of reflection) and backport to lower branches. >> Ultimately, >> release a version of HBase with this feature. >> >> cc: Stephen. >> >> On Wed, Sep 18, 2024 at 12:08 AM Istvan Toth <st...@cloudera.com.invalid> >> wrote: >> >> > Created a new ticket, as the old one was for 3.3.6 but we've agreed on >> > 3.4.0, and expanded the scope. >> > >> > https://issues.apache.org/jira/browse/HBASE-28846 >> > >> > On Tue, Sep 17, 2024 at 3:47 PM 张铎(Duo Zhang) <palomino...@gmail.com> >> > wrote: >> > >> > > I think we could start the work from branch-2.6? Branch-2.5 should >> > > reach its EOL soon, after 2.6.x is stable enough. >> > > >> > > In this way we only need to deal with 3.3.x and 3.4.x. >> > > >> > > Istvan Toth <st...@cloudera.com.invalid> 于2024年9月17日周二 16:56写道： >> > > > >> > > > Thanks for the assessment, Wei-Chiu. >> > > > >> > > > Transitive dependency updates in Hadoop are normal (desired even), >> > that's >> > > > something that HBase needs to manage. >> > > > >> > > > As for the test: >> > > > >> > > > - Duo's suggestion is to extend the Hadoop compatibility tests, and >> run >> > > > them with multiple Hadoop 3 releases. >> > > > Looking at the nightly results, those tests are fast, it takes 14 >> > minutes >> > > > for Hadoop2 and Hadoop3. >> > > > I've peeked into hbase_nightly_pseudo-distributed-test.sh , the >> tests >> > > there >> > > > are indeed quite minimal, >> > > > more of a smoke test, and seem to be targeted to check the shaded >> > > artifacts. >> > > > >> > > > - Nick's suggestion is to run DevTests and set the Hadoop version. >> > > > The runAllTests step in the current nightly takes 8+ hours. >> > > > On my 6+8 core laptop, my last attempt failed after 90 minutes, so >> > let's >> > > > say the full run takes 120 minutes. >> > > > >> > > > I don't know how many free resources HBase has, but if we can >> utilize >> > > > another VM per branch we could run the dev tests with four HBase >> > > versions, >> > > > and still finish about the same time as the full test job does. >> > > > >> > > > We don't need to test with the default version, as we already run >> the >> > > full >> > > > suite for that one. >> > > > >> > > > Assuming that we officially support 3.4.0 on all active branches, >> and >> > > also >> > > > default to 3.4.0 on all branches, and trusting Hadoop's >> compatibility >> > so >> > > > that we don't need to test interim >> > > > patch releases within a minor version, we could go with these >> versions: >> > > > >> > > > branch-2.5 : 3.2.3, 3.2.4, 3.3.2, 3.3.6 >> > > > branch-2.6 : 3.3.5, 3.3.6 >> > > > branch-3: 3.3.5, 3.3.6 >> > > > branch-4: 3.3.5, 3.3.6 >> > > > >> > > > If we trust Hadoop not to break compatibility in patch releases, we >> > > could >> > > > reduce this to only the oldest patch releases: >> > > > >> > > > branch-2.5 : 3.2.3, 3.3.2 >> > > > branch-2.6 : 3.3.5 >> > > > branch-3: 3.3.5 >> > > > branch-4: 3.3.5 >> > > > >> > > > or if we trust it not break compatibility in specific minor >> versions, >> > we >> > > > could further reduce it to just the oldest supported release: >> > > > >> > > > branch-2.5 : 3.2.3 >> > > > branch-2.6 : 3.3.5 >> > > > branch-3: 3.3.5 >> > > > branch-4: 3.3.5 >> > > > >> > > > Of course running every devTest is an overkill, as the vast >> majority of >> > > the >> > > > tests use the same set of Hadoop APIs and features, and we'd only >> > really >> > > > need to run the tests that cover that feature set. >> > > > Figuring out a subset of tests that exercise the full Hadoop API >> (that >> > we >> > > > use) is a hard and error prone task, so if we have the resources, we >> > can >> > > > just brute force it with devTests. >> > > > >> > > > As a base for further discussion: >> > > > >> > > > Let's take the first (first and last supported patch level for each >> > minor >> > > > release) set of versions, >> > > > and run both the pseudistributed tests and the devTests on them. >> > > > >> > > > Does that sound good ? Do we have the resources for that ? Do we >> have a >> > > > better idea ? >> > > > >> > > > Istvan >> > > > >> > > > On Mon, Sep 16, 2024 at 7:20 PM Wei-Chiu Chuang <weic...@apache.org >> > >> > > wrote: >> > > > >> > > > > I strive to meet that stated compatibility goal when I release >> > Hadoop. >> > > > > But we don't have a rigorous compatibility/upgrade test in Hadoop >> so >> > > YMMV >> > > > > (we now have in Ozone!) >> > > > > >> > > > > There are so many gotchas that it really depends on the RM to do >> the >> > > > > hardwork, checking protobuf definitions, running API compat >> report, >> > > > > compiling against downstream applications. >> > > > > The other thing is thirdparty dependency update. Whenever I bump >> > Netty >> > > or >> > > > > Jetty version, new transitive dependencies slip in as part of the >> > > update, >> > > > > which sometimes break HBase because of the dependency check in >> > shading. >> > > > > >> > > > > On Mon, Sep 16, 2024 at 4:48 AM Istvan Toth >> > <st...@cloudera.com.invalid >> > > > >> > > > > wrote: >> > > > > >> > > > > > On Wed, Sep 11, 2024 at 4:30 PM 张铎(Duo Zhang) < >> > palomino...@gmail.com >> > > > >> > > > > > wrote: >> > > > > > >> > > > > > > There is a problem that, usually, you can use an old hadoop >> > client >> > > to >> > > > > > > communicate with a new hadoop server, but not vice versa. >> > > > > > > >> > > > > > >> > > > > > Do we have examples of that ? >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > >> > >> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html >> > > > > > specifically states otherwise: >> > > > > > >> > > > > > In addition to the limitations imposed by being Stable >> > > > > > < >> > > > > > >> > > > > >> > > >> > >> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/InterfaceClassification.html#Stable >> > > > > > >, >> > > > > > Hadoop’s wire protocols MUST also be forward compatible across >> > minor >> > > > > > releases within a major version according to the following: >> > > > > > >> > > > > > - Client-Server compatibility MUST be maintained so as to >> allow >> > > users >> > > > > to >> > > > > > continue using older clients even after upgrading the server >> > > (cluster) >> > > > > > to a >> > > > > > later version (or vice versa). For example, a Hadoop 2.1.0 >> > client >> > > > > > talking >> > > > > > to a Hadoop 2.3.0 cluster. >> > > > > > - Client-Server compatibility MUST be maintained so as to >> allow >> > > users >> > > > > to >> > > > > > upgrade the client before upgrading the server (cluster). For >> > > > > example, a >> > > > > > Hadoop 2.4.0 client talking to a Hadoop 2.3.0 cluster. This >> > allows >> > > > > > deployment of client-side bug fixes ahead of full cluster >> > > upgrades. >> > > > > Note >> > > > > > that new cluster features invoked by new client APIs or shell >> > > commands >> > > > > > will >> > > > > > not be usable. YARN applications that attempt to use new APIs >> > > > > (including >> > > > > > new fields in data structures) that have not yet been >> deployed >> > to >> > > the >> > > > > > cluster can expect link exceptions. >> > > > > > - Client-Server compatibility MUST be maintained so as to >> allow >> > > > > > upgrading individual components without upgrading others. For >> > > example, >> > > > > > upgrade HDFS from version 2.1.0 to 2.2.0 without upgrading >> > > MapReduce. >> > > > > > - Server-Server compatibility MUST be maintained so as to >> allow >> > > mixed >> > > > > > versions within an active cluster so the cluster may be >> upgraded >> > > > > without >> > > > > > downtime in a rolling fashion. >> > > > > > >> > > > > > Admittedly, I don't have a lot of experience with mismatched >> Hadoop >> > > > > > versions, but my proposal should be covered by the second >> clause. >> > > > > > >> > > > > > Usage of newer APIs should be caught when compiling with older >> > Hadoop >> > > > > > versions. >> > > > > > The only risk I can see is when we use a new feature which was >> > added >> > > > > > without changing the API signature (such as adding a new >> constant >> > > value >> > > > > for >> > > > > > some new behaviour) >> > > > > > >> > > > > > >> > > > > > > When deploying HBase, HBase itself acts as a client of hadoop, >> > > that's >> > > > > > > why we always stay on the oldest support hadoop version. >> > > > > > > >> > > > > > > >> > > > > > Not true for 2.6 , which according to the docs supports Hadoop >> 3.2, >> > > but >> > > > > > defaults to Hadoop 3.3 >> > > > > > >> > > > > > >> > > > > > > For me, technically I think bumping to the newest patch >> release >> > of >> > > a >> > > > > > > minor release should be fine, which is the proposal 1. >> > > > > > > >> > > > > > > But the current hadoopcheck is not enough, since it can only >> > ensure >> > > > > > > that there is no complation error. >> > > > > > > Maybe we should also run some simple dev tests in the >> hadoopcheck >> > > > > > > stage, and in integration tests, we should try to build with >> all >> > > the >> > > > > > > support hadoop version and run the basic read write tests. >> > > > > > >> > > > > > >> > > > > > Do we need to test all versions ? >> > > > > > If We test with say, 3.3.0 and 3.3.6 , do we need to test with >> > > 3.3.[1-5] >> > > > > ? >> > > > > > Or if we test with 3.2.5 and 3.3.6, do we need to test with >> any of >> > > the >> > > > > > interim versions ? >> > > > > > >> > > > > > Basically, how much do we trust Hadoop to keep to its >> compatibility >> > > > > rules ? >> > > > > > >> > > > > > Running a limited number of tests should not be a problem. >> > > > > > Should we add a new test category, so that they can be easily >> > started >> > > > > from >> > > > > > Maven ? >> > > > > > >> > > > > > Can you suggest some tests that we should run for the >> compatibility >> > > > > check ? >> > > > > > >> > > > > > >> > > > > > > Thanks. >> > > > > > > >> > > > > > > Istvan Toth <st...@cloudera.com.invalid> 于2024年9月11日周三 >> 21:05写道： >> > > > > > > > >> > > > > > > > Let me summarize my take of the discussion so far: >> > > > > > > > >> > > > > > > > There are two aspects to the HBase version we build with: >> > > > > > > > 1. Source code quality/compatibility >> > > > > > > > 2. Security and usability of the public binary assemblies >> and >> > > > > (shaded) >> > > > > > > > hbase maven artifacts. >> > > > > > > > >> > > > > > > > 1. Source code quality/compatibility >> > > > > > > > >> > > > > > > > AFAICT we have the following hard goals: >> > > > > > > > 1.a : Ensure that HBase compiles and runs well with the >> earlier >> > > > > > supported >> > > > > > > > Hadoop version on the given branch >> > > > > > > > 1.b: Ensure that HBase compiles and runs well with the >> latest >> > > > > supported >> > > > > > > > Hadoop version on the given branch >> > > > > > > > >> > > > > > > > In my opinion we should also strive for these goals: >> > > > > > > > 1.c: Aim to officially support the newest possible Hadoop >> > > releases >> > > > > > > > 1.d: Take advantage of new features in newer Hadoop >> versions >> > > > > > > > >> > > > > > > > 2. Public binary usability wish list: >> > > > > > > > >> > > > > > > > 2.a: We want them to work OOB for as many use cases as >> possible >> > > > > > > > 2.b: We want to work them as well as possible >> > > > > > > > 2.c: We want to have as few CVEs in them as possible >> > > > > > > > 2.d: We want to make upgrades as painless as possible, >> > > especially for >> > > > > > > patch >> > > > > > > > releases >> > > > > > > > >> > > > > > > > The factor that Hadoop does not have an explicit end-of-life >> > > policy >> > > > > of >> > > > > > > > course complicates things. >> > > > > > > > >> > > > > > > > Our current policy seems to be that we pick a Hadoop >> version to >> > > build >> > > > > > > with >> > > > > > > > when releasing a minor version, >> > > > > > > > and stay on that version until there is a newer patch >> released >> > of >> > > > > that >> > > > > > > > minor version with direct CVE fixes. >> > > > > > > > This does not seem to be an absolute, for example the >> recently >> > > > > released >> > > > > > > > HBase 2.4.18 still defaults to Hadoop 3.1.2, >> > > > > > > > which has several old CVEs, many of which are reportedly >> fixed >> > in >> > > > > 3.1.3 >> > > > > > > and >> > > > > > > > 3.1.4. >> > > > > > > > >> > > > > > > > my proposals are : >> > > > > > > > >> > > > > > > > Proposal 1: >> > > > > > > > >> > > > > > > > Whenever a new Hadoop patch release is released for a minor >> > > version, >> > > > > > then >> > > > > > > > unless it breaks source compatibility, we should >> automatically >> > > update >> > > > > > the >> > > > > > > > default Hadoop version for >> > > > > > > > all branches that use the same minor version. >> > > > > > > > The existing hadoopcheck mechanism should be good enough to >> > > guarantee >> > > > > > > that >> > > > > > > > we do not break compatibility with the earlier patch >> releases. >> > > > > > > > >> > > > > > > > This would ensure that the binaries use the latest and >> greatest >> > > > > Hadoop >> > > > > > > (of >> > > > > > > > that minor branch) and that users of the binaries get the >> > latest >> > > > > fixes, >> > > > > > > > both CVE and functionality wise, and >> > > > > > > > the binaries also get the transitive CVE fixes in that >> release. >> > > > > > > > For example,if we did this we could use the new feature in >> > > 3.3.6 in >> > > > > > > > HBASE-27769 (via reflection) and also test it, thereby >> > improving >> > > > > Ozone >> > > > > > > > support. >> > > > > > > > >> > > > > > > > On the other hand we minimize changes and maximize >> > compatibility >> > > by >> > > > > > > > sticking to the same Hadoop minor release. >> > > > > > > > >> > > > > > > > Proposal 2: >> > > > > > > > >> > > > > > > > We should default to the latest hadoop version (currently >> > 3.4.0) >> > > on >> > > > > > > > unreleased branches. >> > > > > > > > This should ensure that when we do release we default to the >> > > latest >> > > > > > > > version, and we've tested it as thoroughly as possible. >> > > > > > > > >> > > > > > > > Again. the existing Hadoopcheck mechanism should ensure >> that we >> > > do >> > > > > not >> > > > > > > > break compatibility with earlier supported versions. >> > > > > > > > >> > > > > > > > Istvan >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Mon, Sep 9, 2024 at 9:41 PM Nick Dimiduk < >> > ndimi...@apache.org >> > > > >> > > > > > wrote: >> > > > > > > > >> > > > > > > > > Yes, we’ll use reflection to make use of APIs introduced >> in >> > > newer >> > > > > > HDFS >> > > > > > > > > versions than the stated dependency until the stated >> > dependency >> > > > > > finally >> > > > > > > > > catches up. >> > > > > > > > > >> > > > > > > > > On Mon, 9 Sep 2024 at 19:55, Wei-Chiu Chuang < >> > > weic...@apache.org> >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Reflection is probably the way to go to ensure maximum >> > > > > > compatibility >> > > > > > > TBH >> > > > > > > > > > >> > > > > > > > > > On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth >> > > > > > > <st...@cloudera.com.invalid> >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Stephen Wu has kindly sent me the link for the >> previous >> > > email >> > > > > > > thread: >> > > > > > > > > > > >> > > > > https://lists.apache.org/thread/2k4tvz3wpg06sgkynkhgvxrodmj86vsj >> > > > > > > > > > > >> > > > > > > > > > > Reading it, I cannot see anything there that would >> > > > > contraindicate >> > > > > > > > > > upgrading >> > > > > > > > > > > to 3.3.6 from 3.3.5, at least on the branches that >> > already >> > > > > > default >> > > > > > > to >> > > > > > > > > > > 3.3.5, i.e. 2.6+. >> > > > > > > > > > > >> > > > > > > > > > > At first glance, the new logic in HBASE-27769 could >> also >> > be >> > > > > > > implemented >> > > > > > > > > > > with the usual reflection hacks, while preserving the >> old >> > > logic >> > > > > > for >> > > > > > > > > > Hadoop >> > > > > > > > > > > 3.3.5 and earlier. >> > > > > > > > > > > >> > > > > > > > > > > Thanks, >> > > > > > > > > > > Istvan >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > On Mon, Sep 9, 2024 at 1:42 PM Istvan Toth < >> > > st...@cloudera.com >> > > > > > >> > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Thanks for your reply, Nick. >> > > > > > > > > > > > >> > > > > > > > > > > > There are no listed direct CVEs in either Hadoop >> 3.2.4 >> > or >> > > > > > 3.3.5, >> > > > > > > but >> > > > > > > > > > > there >> > > > > > > > > > > > are CVEs in their transitive dependencies. >> > > > > > > > > > > > >> > > > > > > > > > > > My impression is that rather than shipping the >> oldest >> > > 'safe' >> > > > > > > version, >> > > > > > > > > > > > HBase does seem to update the default Hadoop >> version to >> > > the >> > > > > > > > > latest-ish >> > > > > > > > > > at >> > > > > > > > > > > > the time of the start >> > > > > > > > > > > > of the release process, otherwise 2.6 would still >> > > default to >> > > > > > > 3.2.4. >> > > > > > > > > > > (HBase >> > > > > > > > > > > > 2.6 release was already underway when Hadoop 3.4.0 >> was >> > > > > > released) >> > > > > > > > > > > > >> > > > > > > > > > > > For now, we (Phoenix) have resorted to dependency >> > > managing >> > > > > > > transitive >> > > > > > > > > > > > dependencies coming in (only) via Hadoop in Phoenix, >> > > > > > > > > > > > but that is a slippery slope, and adds a layer of >> > > > > uncertainty, >> > > > > > > as it >> > > > > > > > > > may >> > > > > > > > > > > > introduce incompatibilities in Hadoop that we don't >> > have >> > > > > tests >> > > > > > > for. >> > > > > > > > > > > > >> > > > > > > > > > > > Our situation is similar to that of the HBase shaded >> > > > > artifacts, >> > > > > > > where >> > > > > > > > > > we >> > > > > > > > > > > > ship a huge uberjar that includes much of both HBase >> > and >> > > > > Hadoop >> > > > > > > on >> > > > > > > > > top >> > > > > > > > > > of >> > > > > > > > > > > > (or rather below) Phoenix, >> > > > > > > > > > > > similar to the hbase-client-shaded jar. >> > > > > > > > > > > > >> > > > > > > > > > > > I will look into to hadoop check CI tests that >> you've >> > > > > > mentioned, >> > > > > > > > > then I >> > > > > > > > > > > > will try to resurrect HBASE-27931, and if I don't >> find >> > > any >> > > > > > > issues, >> > > > > > > > > and >> > > > > > > > > > > > there are no objections, then >> > > > > > > > > > > > I will put a PR to update the unreleased version to >> > > default >> > > > > to >> > > > > > > 3.4.0. >> > > > > > > > > > > > >> > > > > > > > > > > > Istvan >> > > > > > > > > > > > >> > > > > > > > > > > > On Mon, Sep 9, 2024 at 11:06 AM Nick Dimiduk < >> > > > > > > ndimi...@apache.org> >> > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > >> My understanding of our hadoop dependency policy is >> > > that we >> > > > > > ship >> > > > > > > > > poms >> > > > > > > > > > > with >> > > > > > > > > > > >> hadoop versions pinned to the oldest compatible, >> > "safe" >> > > > > > version >> > > > > > > that >> > > > > > > > > > is >> > > > > > > > > > > >> supported. Our test infrastructure has a "hadoop >> > check" >> > > > > > > procedure >> > > > > > > > > that >> > > > > > > > > > > >> does >> > > > > > > > > > > >> some validation against other patch release >> versions. >> > > > > > > > > > > >> >> > > > > > > > > > > >> I don't know if anyone has done a CVE sweep >> recently. >> > If >> > > > > there >> > > > > > > are >> > > > > > > > > new >> > > > > > > > > > > >> CVEs, we do bump the minimum supported version >> > > specified in >> > > > > > the >> > > > > > > pom >> > > > > > > > > as >> > > > > > > > > > > >> part >> > > > > > > > > > > >> of patch releases. These changes need to include a >> > > pretty >> > > > > > > thorough >> > > > > > > > > > > >> compatibility check so that we can include release >> > notes >> > > > > about >> > > > > > > any >> > > > > > > > > > > >> introduced incompatibilities. >> > > > > > > > > > > >> >> > > > > > > > > > > >> I am in favor of a dependency bump so as to address >> > > known >> > > > > CVEs >> > > > > > > as >> > > > > > > > > best >> > > > > > > > > > > as >> > > > > > > > > > > >> we reasonably can. >> > > > > > > > > > > >> >> > > > > > > > > > > >> Thanks, >> > > > > > > > > > > >> Nick >> > > > > > > > > > > >> >> > > > > > > > > > > >> On Mon, Sep 9, 2024 at 10:59 AM Istvan Toth < >> > > > > st...@apache.org >> > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > > > >> >> > > > > > > > > > > >> > Hi! >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > I'm working on building the Phoenix uberjars with >> > > newer >> > > > > > Hadoop >> > > > > > > > > > > versions >> > > > > > > > > > > >> by >> > > > > > > > > > > >> > default to improve its CVE stance, and I realized >> > that >> > > > > HBase >> > > > > > > > > itself >> > > > > > > > > > > does >> > > > > > > > > > > >> > not use the latest releases. >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > branch-2.5 defaults to 3.2.4 >> > > > > > > > > > > >> > branch-2.6 and later defaults to 3.3.5 >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > I can kind of understand that we don't want to >> bump >> > > the >> > > > > > minor >> > > > > > > > > > version >> > > > > > > > > > > >> for >> > > > > > > > > > > >> > branch-2.5 from the one it was released with. >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > However, I don't see the rationale for not >> upgrading >> > > > > > > branch-2.6 to >> > > > > > > > > > at >> > > > > > > > > > > >> least >> > > > > > > > > > > >> > 3.3.6, and the unreleased branches (branch-2, >> > > branch-3, >> > > > > > > master) to >> > > > > > > > > > > >> 3.4.0. >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > I found a mention of wanting to stay off the >> latest >> > > patch >> > > > > > > release >> > > > > > > > > > > >> > HBASE-27931, but I could not figure if it has a >> > > technical >> > > > > > > reason, >> > > > > > > > > or >> > > > > > > > > > > if >> > > > > > > > > > > >> > this is a written (or unwritten) policy. >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > best regards >> > > > > > > > > > > >> > Istvan >> > > > > > > > > > > >> > >> > > > > > > > > > > >> >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > -- >> > > > > > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > > > > > > > > > > > *Email*: st...@cloudera.com >> > > > > > > > > > > > cloudera.com <https://www.cloudera.com> >> > > > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > > > > > > > > > > > [image: Cloudera on Twitter] < >> > > https://twitter.com/cloudera> >> > > > > > > [image: >> > > > > > > > > > > > Cloudera on Facebook] < >> > https://www.facebook.com/cloudera >> > > > >> > > > > > > [image: >> > > > > > > > > > > > Cloudera on LinkedIn] < >> > > > > > https://www.linkedin.com/company/cloudera >> > > > > > > > >> > > > > > > > > > > > ------------------------------ >> > > > > > > > > > > > ------------------------------ >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > -- >> > > > > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > > > > > > > > > > *Email*: st...@cloudera.com >> > > > > > > > > > > cloudera.com <https://www.cloudera.com> >> > > > > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > > > > > > > > > > [image: Cloudera on Twitter] < >> > https://twitter.com/cloudera >> > > > >> > > > > > > [image: >> > > > > > > > > > > Cloudera on Facebook] < >> https://www.facebook.com/cloudera >> > > >> > > > > > [image: >> > > > > > > > > > Cloudera >> > > > > > > > > > > on LinkedIn] < >> https://www.linkedin.com/company/cloudera> >> > > > > > > > > > > ------------------------------ >> > > > > > > > > > > ------------------------------ >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > *István Tóth* | Sr. Staff Software Engineer >> > > > > > > > *Email*: st...@cloudera.com >> > > > > > > > cloudera.com <https://www.cloudera.com> >> > > > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > > > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> >> > > [image: >> > > > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> >> > > [image: >> > > > > > > Cloudera >> > > > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > > > > > > > ------------------------------ >> > > > > > > > ------------------------------ >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > *István Tóth* | Sr. Staff Software Engineer >> > > > > > *Email*: st...@cloudera.com >> > > > > > cloudera.com <https://www.cloudera.com> >> > > > > > [image: Cloudera] <https://www.cloudera.com/> >> > > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> >> > [image: >> > > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> >> [image: >> > > > > Cloudera >> > > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > > > > > ------------------------------ >> > > > > > ------------------------------ >> > > > > > >> > > > > >> > > > >> > > > >> > > > -- >> > > > *István Tóth* | Sr. Staff Software Engineer >> > > > *Email*: st...@cloudera.com >> > > > cloudera.com <https://www.cloudera.com> >> > > > [image: Cloudera] <https://www.cloudera.com/> >> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> > > Cloudera >> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > > > ------------------------------ >> > > > ------------------------------ >> > > >> > >> > >> > -- >> > *István Tóth* | Sr. Staff Software Engineer >> > *Email*: st...@cloudera.com >> > cloudera.com <https://www.cloudera.com> >> > [image: Cloudera] <https://www.cloudera.com/> >> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: >> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: >> Cloudera >> > on LinkedIn] <https://www.linkedin.com/company/cloudera> >> > ------------------------------ >> > ------------------------------ >> > >> > > > -- > *István Tóth* | Sr. Staff Software Engineer > *Email*: st...@cloudera.com > cloudera.com <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> > ------------------------------ > ------------------------------ > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
