[jira] [Created] (TEZ-4565) TestAnalyzer subtest testInternalPreemption is flaky
Jonathan Turner Eagles created TEZ-4565: --- Summary: TestAnalyzer subtest testInternalPreemption is flaky Key: TEZ-4565 URL: https://issues.apache.org/jira/browse/TEZ-4565 Project: Apache Tez Issue Type: Test Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (TEZ-4562) Fix Tez Job Analyzer after TEZ_DAG_EXTRA_INFO
Jonathan Turner Eagles created TEZ-4562: --- Summary: Fix Tez Job Analyzer after TEZ_DAG_EXTRA_INFO Key: TEZ-4562 URL: https://issues.apache.org/jira/browse/TEZ-4562 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles TEZ-3611 split DAG INFO and DAG EXTRA INFO but tez job analyzer wasn't updated to account for the change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (TEZ-4558) Update build setup maven version and enforcer minimum to correct minimum
[ https://issues.apache.org/jira/browse/TEZ-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4558. - Fix Version/s: 0.10.4 Resolution: Fixed > Update build setup maven version and enforcer minimum to correct minimum > > > Key: TEZ-4558 > URL: https://issues.apache.org/jira/browse/TEZ-4558 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Turner Eagles >Assignee: Jonathan Turner Eagles >Priority: Major > Fix For: 0.10.4 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Build can't succeed according to build instructions and maven required > version enforcement. > maven-enforcer-plugin: requireMavenVersion 3.0.2 > [MVNVM] Using maven: 3.1.0 > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce > (enforce-maven-version) on project tez: The plugin > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0 requires Maven version > 3.1.1 -> [Help 1] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TEZ-4558) Update build setup maven version and enforcer minimum to correct minimum
[ https://issues.apache.org/jira/browse/TEZ-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843003#comment-17843003 ] Jonathan Turner Eagles commented on TEZ-4558: - Personally, I've been using 3.6.3. But have been using mvnvm recently which grabs the maven required version from the pom file and uses that maven version to build the project with. > Update build setup maven version and enforcer minimum to correct minimum > > > Key: TEZ-4558 > URL: https://issues.apache.org/jira/browse/TEZ-4558 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Turner Eagles >Assignee: Jonathan Turner Eagles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Build can't succeed according to build instructions and maven required > version enforcement. > maven-enforcer-plugin: requireMavenVersion 3.0.2 > [MVNVM] Using maven: 3.1.0 > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce > (enforce-maven-version) on project tez: The plugin > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0 requires Maven version > 3.1.1 -> [Help 1] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (TEZ-4558) Update build setup maven version and enforcer minimum to correct minimum
[ https://issues.apache.org/jira/browse/TEZ-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated TEZ-4558: Description: Build can't succeed according to build instructions and maven required version enforcement. maven-enforcer-plugin: requireMavenVersion 3.0.2 [MVNVM] Using maven: 3.1.0 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce (enforce-maven-version) on project tez: The plugin org.apache.maven.plugins:maven-enforcer-plugin:3.0.0 requires Maven version 3.1.1 -> [Help 1] > Update build setup maven version and enforcer minimum to correct minimum > > > Key: TEZ-4558 > URL: https://issues.apache.org/jira/browse/TEZ-4558 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Turner Eagles >Assignee: Jonathan Turner Eagles >Priority: Major > > Build can't succeed according to build instructions and maven required > version enforcement. > maven-enforcer-plugin: requireMavenVersion 3.0.2 > [MVNVM] Using maven: 3.1.0 > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce > (enforce-maven-version) on project tez: The plugin > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0 requires Maven version > 3.1.1 -> [Help 1] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (TEZ-4558) Update build setup maven version and enforcer minimum to correct minimum
Jonathan Turner Eagles created TEZ-4558: --- Summary: Update build setup maven version and enforcer minimum to correct minimum Key: TEZ-4558 URL: https://issues.apache.org/jira/browse/TEZ-4558 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (TEZ-4553) Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535
[ https://issues.apache.org/jira/browse/TEZ-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4553. - Fix Version/s: 0.10.4 Resolution: Fixed > Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535 > - > > Key: TEZ-4553 > URL: https://issues.apache.org/jira/browse/TEZ-4553 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Turner Eagles >Assignee: Jonathan Turner Eagles >Priority: Major > Fix For: 0.10.4 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (TEZ-4553) Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535
Jonathan Turner Eagles created TEZ-4553: --- Summary: Default task scheduler to DagAwareTaskScheduler to avoid hang in TEZ-3535 Key: TEZ-4553 URL: https://issues.apache.org/jira/browse/TEZ-4553 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TEZ-2119) Counter for launched containers
[ https://issues.apache.org/jira/browse/TEZ-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758649#comment-17758649 ] Jonathan Turner Eagles commented on TEZ-2119: - I think this looks good. Adding review to github. Good work, [~abstractdog] > Counter for launched containers > --- > > Key: TEZ-2119 > URL: https://issues.apache.org/jira/browse/TEZ-2119 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Rohini Palaniswamy >Assignee: László Bodor >Priority: Major > Attachments: TEZ-2119.01.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > org.apache.tez.common.counters.DAGCounter > NUM_SUCCEEDED_TASKS=32976 > TOTAL_LAUNCHED_TASKS=32976 > OTHER_LOCAL_TASKS=2 > DATA_LOCAL_TASKS=9147 > RACK_LOCAL_TASKS=23761 > It would be very nice to have TOTAL_LAUNCHED_CONTAINERS counter added to > this. The difference between TOTAL_LAUNCHED_CONTAINERS and > TOTAL_LAUNCHED_TASKS should make it easy to see how much container reuse is > happening. It is very hard to find out now. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TEZ-4419) Upgrade node and yarn version and fix npm security issues in Tez UI module
[ https://issues.apache.org/jira/browse/TEZ-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551189#comment-17551189 ] Jonathan Turner Eagles commented on TEZ-4419: - I think many or all of these are solved by TEZ-4337 > Upgrade node and yarn version and fix npm security issues in Tez UI module > -- > > Key: TEZ-4419 > URL: https://issues.apache.org/jira/browse/TEZ-4419 > Project: Apache Tez > Issue Type: Improvement >Reporter: Nikhil Gupta >Assignee: Aman Raj >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (TEZ-4337) [TEZ UI] Upgrade to Ember 3
[
https://issues.apache.org/jira/browse/TEZ-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17430794#comment-17430794
]
Jonathan Turner Eagles commented on TEZ-4337:
-
Documenting the steps for testing puppeteer in ubuntu docker
{code:bash}
% docker build -f build-tools/docker/Dockerfile .
% docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
87e0e291e68f About a minute ago 2.02GB
# Use image id to run new container
docker run -it
npm install -g yarn
adduser me
su me
cd
yarn add puppeteer
cat > run-chrome.js
const puppeteer = require("puppeteer");
const { execFileSync } = require("child_process");
let exePath = puppeteer.executablePath();
let args = process.argv.slice(2);
console.log('exePath', exePath);
console.log('args', args);
execFileSync(exePath, args);
node run-chrome.js --headless --no-sandbox
exit
exit
docker container ls --all
docker rm
docker rmi
{code}
> [TEZ UI] Upgrade to Ember 3
> ---
>
> Key: TEZ-4337
> URL: https://issues.apache.org/jira/browse/TEZ-4337
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4337) [TEZ UI] Upgrade to Ember 3
[ https://issues.apache.org/jira/browse/TEZ-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427948#comment-17427948 ] Jonathan Turner Eagles commented on TEZ-4337: - In addition to upgrading the tez ui component to Ember 3, this jira accomplishes a few other things. * Reduces code size with tree-shaking * Remove ember bootstrap code and dependency and switch to bootstrap less only dependency * Remove jquery dependency and disabled ember optional jquery dependency * Replace moment with date-fns * Upgrade to Node LTS 14, Ember LTS 3.24, Yarn 1.22 * Removed bower dependencies * Convert templates from deprecated dasherized format to Camel case * Upgraded to qunit 5 tests * Upgrade to mode d3 v5. (tree shaking) * Increased security with content security and removal of eval and Function (see exception alasql) * Increased dag loading performance on large dag * Remove tez ui verbose build warnings > [TEZ UI] Upgrade to Ember 3 > --- > > Key: TEZ-4337 > URL: https://issues.apache.org/jira/browse/TEZ-4337 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Turner Eagles >Assignee: Jonathan Turner Eagles >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4337) [TEZ UI] Upgrade to Ember 3
Jonathan Turner Eagles created TEZ-4337: --- Summary: [TEZ UI] Upgrade to Ember 3 Key: TEZ-4337 URL: https://issues.apache.org/jira/browse/TEZ-4337 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4330) Import external tez component em-tgraph
[ https://issues.apache.org/jira/browse/TEZ-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17409022#comment-17409022 ] Jonathan Turner Eagles commented on TEZ-4330: - [~Sreenath] / [~abstractdog], em-helpers and em-table and imported. Is there anything else needed for em-tgraph? > Import external tez component em-tgraph > --- > > Key: TEZ-4330 > URL: https://issues.apache.org/jira/browse/TEZ-4330 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4329) Import external tez component em-table
[ https://issues.apache.org/jira/browse/TEZ-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4329. - Fix Version/s: 0.10.2 0.9.3 Resolution: Fixed Thanks for review. Merged to master and branch-0.9 > Import external tez component em-table > -- > > Key: TEZ-4329 > URL: https://issues.apache.org/jira/browse/TEZ-4329 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Assignee: Sreenath Somarajapuram >Priority: Major > Fix For: 0.9.3, 0.10.2 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TEZ-4329) Import external tez component em-table
[ https://issues.apache.org/jira/browse/TEZ-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles reassigned TEZ-4329: --- Assignee: Sreenath Somarajapuram > Import external tez component em-table > -- > > Key: TEZ-4329 > URL: https://issues.apache.org/jira/browse/TEZ-4329 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Assignee: Sreenath Somarajapuram >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4330) Import external tez component em-tgraph
[ https://issues.apache.org/jira/browse/TEZ-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404657#comment-17404657 ] Jonathan Turner Eagles commented on TEZ-4330: - FYI, [~Sreenath]. [~abstractdog], straight forward code move into our code base. > Import external tez component em-tgraph > --- > > Key: TEZ-4330 > URL: https://issues.apache.org/jira/browse/TEZ-4330 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4329) Import external tez component em-table
[ https://issues.apache.org/jira/browse/TEZ-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404494#comment-17404494 ] Jonathan Turner Eagles commented on TEZ-4329: - [~abstractdog], do you think you could help with this PR? Essentially moved code from external repo to internal repo. > Import external tez component em-table > -- > > Key: TEZ-4329 > URL: https://issues.apache.org/jira/browse/TEZ-4329 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4329) Import external tez component em-table
[ https://issues.apache.org/jira/browse/TEZ-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404155#comment-17404155 ] Jonathan Turner Eagles commented on TEZ-4329: - [~Sreenath], imported em-table into tez-ui with apache license header. > Import external tez component em-table > -- > > Key: TEZ-4329 > URL: https://issues.apache.org/jira/browse/TEZ-4329 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4328) Import external tez component em-helpers
[ https://issues.apache.org/jira/browse/TEZ-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4328. - Fix Version/s: 0.10.2 0.9.3 Resolution: Fixed Thanks, [~Sreenath]. > Import external tez component em-helpers > > > Key: TEZ-4328 > URL: https://issues.apache.org/jira/browse/TEZ-4328 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Fix For: 0.9.3, 0.10.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TEZ-4328) Import external tez component em-helpers
[ https://issues.apache.org/jira/browse/TEZ-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles reassigned TEZ-4328: --- Assignee: Sreenath Somarajapuram > Import external tez component em-helpers > > > Key: TEZ-4328 > URL: https://issues.apache.org/jira/browse/TEZ-4328 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Assignee: Sreenath Somarajapuram >Priority: Major > Fix For: 0.9.3, 0.10.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4328) Import external tez component em-helpers
[ https://issues.apache.org/jira/browse/TEZ-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403499#comment-17403499 ] Jonathan Turner Eagles commented on TEZ-4328: - I will model the other imports off of this one. Hopefully, finishing this week. > Import external tez component em-helpers > > > Key: TEZ-4328 > URL: https://issues.apache.org/jira/browse/TEZ-4328 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4328) Import external tez component em-helpers
[ https://issues.apache.org/jira/browse/TEZ-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403496#comment-17403496 ] Jonathan Turner Eagles commented on TEZ-4328: - If there is any specific needs, I will reach out and communicate it to you soon > Import external tez component em-helpers > > > Key: TEZ-4328 > URL: https://issues.apache.org/jira/browse/TEZ-4328 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4328) Import external tez component em-helpers
[ https://issues.apache.org/jira/browse/TEZ-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403477#comment-17403477 ] Jonathan Turner Eagles commented on TEZ-4328: - [~Sreenath], this is a first import for em-helpers. You don't have to do a full review, but there is some signoff as code owner. I have moved all functionality into the tez ui itself. I have added the Apache license header. As owner for this code it will be important for you to sign off on the license. https://github.com/apache/tez/pull/145/files > Import external tez component em-helpers > > > Key: TEZ-4328 > URL: https://issues.apache.org/jira/browse/TEZ-4328 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4331) Import external tez component mock-ats
Jonathan Turner Eagles created TEZ-4331: --- Summary: Import external tez component mock-ats Key: TEZ-4331 URL: https://issues.apache.org/jira/browse/TEZ-4331 Project: Apache Tez Issue Type: Sub-task Reporter: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4330) Import external tez component em-tgraph
Jonathan Turner Eagles created TEZ-4330: --- Summary: Import external tez component em-tgraph Key: TEZ-4330 URL: https://issues.apache.org/jira/browse/TEZ-4330 Project: Apache Tez Issue Type: Sub-task Reporter: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4329) Import external tez component em-table
[ https://issues.apache.org/jira/browse/TEZ-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated TEZ-4329: Parent: TEZ-4327 Issue Type: Sub-task (was: Bug) > Import external tez component em-table > -- > > Key: TEZ-4329 > URL: https://issues.apache.org/jira/browse/TEZ-4329 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Turner Eagles >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4329) Import external tez component em-table
Jonathan Turner Eagles created TEZ-4329: --- Summary: Import external tez component em-table Key: TEZ-4329 URL: https://issues.apache.org/jira/browse/TEZ-4329 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4328) Import external tez component em-helpers
Jonathan Turner Eagles created TEZ-4328: --- Summary: Import external tez component em-helpers Key: TEZ-4328 URL: https://issues.apache.org/jira/browse/TEZ-4328 Project: Apache Tez Issue Type: Sub-task Reporter: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEZ-4327) Migrate external Tez UI components under apache tez control
Jonathan Turner Eagles created TEZ-4327: --- Summary: Migrate external Tez UI components under apache tez control Key: TEZ-4327 URL: https://issues.apache.org/jira/browse/TEZ-4327 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Turner Eagles External dependencies that are owned by tez but not part of tez git. https://github.com/sreenaths/mock-ats https://github.com/sreenaths/em-helpers https://github.com/sreenaths/em-table https://github.com/sreenaths/em-tgraph We should make a plan regarding importing these into the git repo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4069) Avoid repeated computation of preferred locations in split grouping.
[ https://issues.apache.org/jira/browse/TEZ-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399946#comment-17399946 ] Jonathan Turner Eagles commented on TEZ-4069: - [~abstractdog]/ [~odraese], This patch relies on correct implementation of hashCode and equals for SplitContainers. With this, I'm afraid the optimization will have worse performance than the current implementation. Can we confirm performance increase across other implementations (HostAffinitySplitLocationProvider mentioned above)? Can we confirm default hashCode and equals? > Avoid repeated computation of preferred locations in split grouping. > > > Key: TEZ-4069 > URL: https://issues.apache.org/jira/browse/TEZ-4069 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.2 >Reporter: Oliver Draese >Priority: Major > Attachments: TEZ-4069.1.patch, TEZ-4069.patch > > > The TezSplitGrouper iterates through the list of splits multiple times, when > trying to group the splits (see getGroupedSplits). Each time, it asks the > locationProvider to return the array of preferred locations for the splits. > This has two side effects: > * generating the list of preferred locations can cause some CPU overhead > (i.e. calculating the consistent hash in HostAffinitySplitLocationProvider), > which can be avoided > * if the list of preferred location is changing between the different loops > of getGroupedSplits, we might encounter a NullPointerException. This happens > if a new location appears, that was not part of the initial set of locations > when populating the distinctLocations map. > The getGroupedSplits should query the preferred locations only once (for each > split) via the location provider and then memorize these instead of asking > the location provider repeatedly. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4325) Counter for sort time
[ https://issues.apache.org/jira/browse/TEZ-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398100#comment-17398100 ] Jonathan Turner Eagles commented on TEZ-4325: - These open jiras regarding sort counters have some interesting discussion. Hopefully we can build off of those learnings. > Counter for sort time > - > > Key: TEZ-4325 > URL: https://issues.apache.org/jira/browse/TEZ-4325 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: Counters_SORT_TIME_TPCDS_q40.txt > > Time Spent: 10m > Remaining Estimate: 0h > > It's already logged for every SortSpan in PipelinedSorter, but it can be > exposed via a counter to get better insights quickly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-2262) DAG/Tasks should not fail if counter limits are exceeded.
[
https://issues.apache.org/jira/browse/TEZ-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-2262.
-
Resolution: Incomplete
Creating an option to make counters limits exceeded non-fatal is valid. This
can be achieved by setting a very high limit as a work around. Closing as
incomplete
> DAG/Tasks should not fail if counter limits are exceeded.
> -
>
> Key: TEZ-2262
> URL: https://issues.apache.org/jira/browse/TEZ-2262
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Mostafa Mokhtar
>Priority: Major
>
> Running TPC-DS Q64 failed due to exceeding the max number of counters.
> DAG should succeed and include a warning in the diagnostics stating that the
> error got truncated.
> {code}
> 18043560327-2015-04-01 16:23:08,509 INFO [AsyncDispatcher event handler]
> impl.DAGImpl: No output committers for vertex: Reducer 9
> 18043560445-2015-04-01 16:23:08,857 FATAL [AsyncDispatcher event handler]
> event.AsyncDispatcher: Error in dispatcher thread
> 18043560557:org.apache.tez.common.counters.LimitExceededException: Too many
> counters: 1201 max=1200
> 18043560645- at
> org.apache.tez.common.counters.Limits.checkCounters(Limits.java:87)
> 18043560717- at
> org.apache.tez.common.counters.Limits.incrCounters(Limits.java:94)
> 18043560788- at
> org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:75)
> 18043560885- at
> org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:92)
> 18043560986- at
> org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:103)
> 18043561085- at
> org.apache.tez.common.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:198)
> 18043561188- at
> org.apache.tez.common.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:363)
> 18043561283- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.incrTaskCounters(DAGImpl.java:598)
> 18043561362- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:588)
> 18043561439- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.logJobHistoryFinishedEvent(DAGImpl.java:994)
> 18043561528- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.finished(DAGImpl.java:1135)
> 18043561600- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.checkDAGForCompletion(DAGImpl.java:1048)
> 18043561685- at
> org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1708)
> 18043561785- at
> org.apache.tez.dag.app.dag.impl.DAGImpl$VertexCompletedTransition.transition(DAGImpl.java:1665)
> 18043561885- at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> 18043562001- at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> 18043562097- at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> 18043562190- at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> 18043562307- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:944)
> 18043562376- at
> org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:126)
> 18043562445- at
> org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1686)
> 18043562535- at
> org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:1677)
> 18043562625- at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
> 18043562709- at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
> 18043562790- at java.lang.Thread.run(Thread.java:745)
> 18043562832-2015-04-01 16:23:08,882 INFO [AsyncDispatcher event handler]
> event.AsyncDispatcher: Exiting, bbye..
> 18043562932-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster:
> DAGAppMasterShutdownHook invoked
> 18043563023-2015-04-01 16:23:08,885 INFO [Thread-1] app.DAGAppMaster:
> DAGAppMaster received a signal. Signaling TaskScheduler
> 18043563137-2015-04-01 16:23:08,885 INFO [Thread-1]
> rm.TaskSchedulerEventHandler: TaskScheduler notified that iSignalled was :
> true
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4293) Use Tez credentials for CI
[
https://issues.apache.org/jira/browse/TEZ-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397382#comment-17397382
]
Jonathan Turner Eagles commented on TEZ-4293:
-
Thanks for the patch, [~abstractdog].
> Use Tez credentials for CI
> --
>
> Key: TEZ-4293
> URL: https://issues.apache.org/jira/browse/TEZ-4293
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.9.3, 0.10.2
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> TEZ-4291 only managed to work with hadoop credentials, this needs to be fixed.
> For yetus debugging, this revision can be used:
> https://github.com/abstractdog/yetus/commit/aa8223ecdf84eb7cd963c3e5acb2e31b60a19b97#diff-9cb6a2dcf2678adfcaa19a33ce63447f9545744d22c511a4b2bb3d909dee2bc1R587-R593
> For manual testing:
> {code}
> curl -X POST -H "Accept: application/vnd.github.v3.full+json" -H
> "Content-Type: application/json" -H "Authorization: token ${GITHUB_TOKEN}" -d
> '{"body":"HELLO TEST COMMENT"}' --location
> "https://api.github.com/repos/apache/tez/issues/112/comments;
> {code}
> Last error message which is shown from jenkins with the logging yetus
> revision is:
> {code}
> 23:56:30
>
> 23:56:30
>
> 23:56:30 Adding comment to Github
> 23:56:30
>
> 23:56:30
>
> 23:56:30
> 23:56:30
> 23:56:30 authenticating with token
> 23:56:30 POSTING TO
> https://api.github.com/repos/apache/tez/issues/112/comments
> 23:56:30% Total% Received % Xferd Average Speed TimeTime
> Time Current
> 23:56:30 Dload Upload Total Spent
> Left Speed
> 23:56:30
> 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
> 100 19180 0 100 1918 0 8678 --:--:-- --:--:-- --:--:-- 8639
> 100 2072 100 154 100 1918603 7521 --:--:-- --:--:-- --:--:-- 8093
> 23:56:30 {
> 23:56:30"message": "Resource not accessible by integration",
> 23:56:30"documentation_url":
> "https://docs.github.com/rest/reference/issues#create-an-issue-comment;
> 23:56:30 }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TEZ-4321) Do Not Wrap Singleton Collections
[
https://issues.apache.org/jira/browse/TEZ-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-4321.
-
Fix Version/s: 0.10.2
0.9.3
Resolution: Fixed
Thanks, [~belugabehr]. Simple straight-forward improvement.
> Do Not Wrap Singleton Collections
> -
>
> Key: TEZ-4321
> URL: https://issues.apache.org/jira/browse/TEZ-4321
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 0.9.3, 0.10.2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Singleton Collections are by their design immutable. There is no need to
> wrap them in an unmodifiableCollection.
> https://docs.oracle.com/javase/9/docs/api/java/util/Collections.html#singletonList-T-
> {code:none}
> public static List singletonList(T o)
> Returns an immutable list containing only the specified object.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TEZ-4314) Fix some plugin versions in site generation
[
https://issues.apache.org/jira/browse/TEZ-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-4314.
-
Fix Version/s: 0.10.2
0.9.3
Resolution: Fixed
> Fix some plugin versions in site generation
> ---
>
> Key: TEZ-4314
> URL: https://issues.apache.org/jira/browse/TEZ-4314
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.9.3, 0.10.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code}
> mvn site
> {code}
> command failed for me with various errors, which I fixed as below:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7.1:site (default-site) on
> project tez-docs: failed to get report for
> org.apache.maven.plugins:maven-project-info-reports-plugin: Could not find
> goal 'project-team' in plugin
> org.apache.maven.plugins:maven-project-info-reports-plugin:3.0.0 among
> available goals ci-management, dependencies, dependency-convergence,
> dependency-info, dependency-management, distribution-management, help, index,
> issue-management, licenses, mailing-lists, modules, plugin-management,
> plugins, scm, summary, team -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :tez-docs
> {code}
> 1. maven-site-plugin
> 3.x doesn't like some html elements within XML, instead of CDATA and other
> hacks, I fixed with an old plugin version to pass for now
> error was like:
> {code}
> Caused by: org.codehaus.plexus.util.xml.pull.XmlPullParserException: TEXT
> must be immediately followed by END_TAG and not START_TAG (position:
> START_TAG seen ...\n End of Google analytics -->\n
>
[jira] [Resolved] (TEZ-4267) Remove Superfluous Code from DAGAppMaster
[ https://issues.apache.org/jira/browse/TEZ-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4267. - Fix Version/s: 0.10.2 0.9.3 Resolution: Fixed +1. Thanks, [~belugabehr]. Committed this change and cherry-picked to branch-0.9 with a few minor conflicts. > Remove Superfluous Code from DAGAppMaster > - > > Key: TEZ-4267 > URL: https://issues.apache.org/jira/browse/TEZ-4267 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 0.9.3, 0.10.2 > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4313) Apache Tez Release 0.10.1
[ https://issues.apache.org/jira/browse/TEZ-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated TEZ-4313: Fix Version/s: (was: 0.10.2) 0.10.1 > Apache Tez Release 0.10.1 > - > > Key: TEZ-4313 > URL: https://issues.apache.org/jira/browse/TEZ-4313 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4313) Apache Tez Release 0.10.1
[ https://issues.apache.org/jira/browse/TEZ-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4313. - Fix Version/s: 0.10.2 Resolution: Fixed I'm going to assume the failures are ok in this particular case. > Apache Tez Release 0.10.1 > - > > Key: TEZ-4313 > URL: https://issues.apache.org/jira/browse/TEZ-4313 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4308) Add Whitespace in ShuffleScheduler Error Message
[ https://issues.apache.org/jira/browse/TEZ-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4308. - Fix Version/s: 0.10.2 0.9.3 Resolution: Fixed > Add Whitespace in ShuffleScheduler Error Message > > > Key: TEZ-4308 > URL: https://issues.apache.org/jira/browse/TEZ-4308 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Fix For: 0.9.3, 0.10.2 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4275) Use Google Guava Intern Facility
[ https://issues.apache.org/jira/browse/TEZ-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4275. - Fix Version/s: 0.10.2 0.9.3 Resolution: Fixed Thanks, [~belugabehr]. Committed to master and branch-0.9. > Use Google Guava Intern Facility > > > Key: TEZ-4275 > URL: https://issues.apache.org/jira/browse/TEZ-4275 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 0.9.3, 0.10.2 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Google Guava has a pretty cool Interner facility. > > * More memory efficient than current offering. Map contains a weak key and > a static dummy value (current implementation uses a weak value) > * Current implementation has a single lock around the entire data structure. > Google segments their data structure into (default: 4) segments for better > concurrency > * All the other thoughtful stuff Google has added into this feature -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4298) ShuffleHandler is not source compatible with Hadoop 3.3.0
[
https://issues.apache.org/jira/browse/TEZ-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17369677#comment-17369677
]
Jonathan Turner Eagles commented on TEZ-4298:
-
[~tasanuma], I have pushed this change to branch-0.9. There is nothing specific
to hadoop 3.3 about this change and it will help support both lines as it will
be easier to back port other changes to branch-0.9 without conflict.
As far as running tez 0.9 releases on hadoop 3.3, that is problematic. There
are a number conflicting dependencies with hadoop 3.3 that prevent hadoop 3.3
and tez 0.9 being in the same classpath (jetty, protobuf) during runtime. Tez
0.9 and 0.10 are nearly identical with the most significant change being hadoop
3.x compatibility.
> ShuffleHandler is not source compatible with Hadoop 3.3.0
> -
>
> Key: TEZ-4298
> URL: https://issues.apache.org/jira/browse/TEZ-4298
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Fix For: 0.9.3, 0.10.2
>
> Attachments: TEZ-4298.001.patch, TEZ-4298.001.patch,
> TEZ-4298.002.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {noformat}
> mvn clean package -DskipTests -pl '!tez-ui' -Dhadoop.version=3.3.0
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[764,45]
> incompatible types: com.google.protobuf.ByteString cannot be converted to
> org.apache.hadoop.thirdparty.protobuf.ByteString
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TEZ-4298) ShuffleHandler is not source compatible with Hadoop 3.3.0
[
https://issues.apache.org/jira/browse/TEZ-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated TEZ-4298:
Fix Version/s: 0.9.3
> ShuffleHandler is not source compatible with Hadoop 3.3.0
> -
>
> Key: TEZ-4298
> URL: https://issues.apache.org/jira/browse/TEZ-4298
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Fix For: 0.9.3, 0.10.2
>
> Attachments: TEZ-4298.001.patch, TEZ-4298.001.patch,
> TEZ-4298.002.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {noformat}
> mvn clean package -DskipTests -pl '!tez-ui' -Dhadoop.version=3.3.0
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[764,45]
> incompatible types: com.google.protobuf.ByteString cannot be converted to
> org.apache.hadoop.thirdparty.protobuf.ByteString
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4298) ShuffleHandler is not source compatible with Hadoop 3.3.0
[
https://issues.apache.org/jira/browse/TEZ-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17367670#comment-17367670
]
Jonathan Turner Eagles commented on TEZ-4298:
-
+1. From me on 002 patch changes. I will approve PR and merge.
> ShuffleHandler is not source compatible with Hadoop 3.3.0
> -
>
> Key: TEZ-4298
> URL: https://issues.apache.org/jira/browse/TEZ-4298
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Attachments: TEZ-4298.001.patch, TEZ-4298.001.patch,
> TEZ-4298.002.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {noformat}
> mvn clean package -DskipTests -pl '!tez-ui' -Dhadoop.version=3.3.0
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[764,45]
> incompatible types: com.google.protobuf.ByteString cannot be converted to
> org.apache.hadoop.thirdparty.protobuf.ByteString
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4297) Hive CLI not working after upgrading from Oracle JDK 8u112 to 8u281 and have errors with TEZ
[
https://issues.apache.org/jira/browse/TEZ-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331068#comment-17331068
]
Jonathan Turner Eagles commented on TEZ-4297:
-
The code above is all part of hive and hadoop, so I think there is no Tez issue
here. [~norbert.k.b.maclang], is this still a problem? From my looking at the
log above, it seems there is a misconfiguration of KMS.
> Hive CLI not working after upgrading from Oracle JDK 8u112 to 8u281 and have
> errors with TEZ
>
>
> Key: TEZ-4297
> URL: https://issues.apache.org/jira/browse/TEZ-4297
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Norbert Kiam Maclang
>Priority: Major
>
> After upgrading Oracle JDK version from jdk-8u112 to jdk-8u281, Hive CLI is
> not working anymore and gives below error when logging in.
> {code:java}
> WARNING: Use "yarn jar" to launch YARN applications.
> 21/03/09 11:00:04 WARN conf.HiveConf: HiveConf of name
> hive.server2.enable.impersonation does not existLogging initialized using
> configuration in file:/etc/hive/2.4.3.0-227/0/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: java.io.IOException:
> Previous writer likely failed to write
> hdfs://ppcontent-nn1.pp-content.dataplatform.com:8020/tmp/hive/hive/_tez_session_dir/96b21825-63f4-4316-9c43-20ebe641d9c9/hive-hcatalog-core.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:544)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: Previous writer likely failed to write
> hdfs://ppcontent-nn1.pp-content.dataplatform.com:8020/tmp/hive/hive/_tez_session_dir/96b21825-63f4-4316-9c43-20ebe641d9c9/hive-hcatalog-core.jar.
> Failing because I am unlikely to write too.
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:982)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:862)
> at
> org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:805)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:233)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:158)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:117)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:541)
> ... 8 more
> {code}
> Version we are using:
> * Ambari 2.2.2
> * Hive 1.2.1
> * Hadoop 2.7
> * Spark 1.6
> * HDP 2.4
> * Tez 0.7.0.2.4
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TEZ-3987) Schedule giving priorities based on topological order
[
https://issues.apache.org/jira/browse/TEZ-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-3987.
-
Resolution: Later
If this feature is still needed in addition to TEZ-394, we can reopen.
> Schedule giving priorities based on topological order
> -
>
> Key: TEZ-3987
> URL: https://issues.apache.org/jira/browse/TEZ-3987
> Project: Apache Tez
> Issue Type: New Feature
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>
> It'd be an improvement for some DAGs to be scheduled in a topological order
> as opposed to the scheduling based on distance from the root from
> {{DAGScheduler}} and {{DAGSchedulerControlled}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4245) Optimise split grouping when locality information is set to null/empty
[ https://issues.apache.org/jira/browse/TEZ-4245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331064#comment-17331064 ] Jonathan Turner Eagles commented on TEZ-4245: - [~rajesh.balamohan], is this patch ready for review? > Optimise split grouping when locality information is set to null/empty > -- > > Key: TEZ-4245 > URL: https://issues.apache.org/jira/browse/TEZ-4245 > Project: Apache Tez > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > Attachments: TEZ-4245.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > In objectstores like S3, locality information always shows up as "localhost". > Having this information in inputsplit slows down scheduling as explained in > https://issues.apache.org/jira/browse/HIVE-14060 Systems like hive remove > "localhost" information from splits. > > Split information without any locality information (localhost/null/empty) > should be treated equally, so that split grouping can do meaningful grouping > based on cluster size. This is to avoid creating small split groups, which > can significantly increase runtime due to sequential processing (i.e same map > task getting lots of inputs and system ends up spending time in > open/seek/close on objectstores). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4192) Attach task specific info in scheduler and retrieve in communicator
[ https://issues.apache.org/jira/browse/TEZ-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331061#comment-17331061 ] Jonathan Turner Eagles commented on TEZ-4192: - [~mustafaiman], this patch seems to be coming along. Are you looking for a review? > Attach task specific info in scheduler and retrieve in communicator > --- > > Key: TEZ-4192 > URL: https://issues.apache.org/jira/browse/TEZ-4192 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.2 >Reporter: Mustafa İman >Assignee: Mustafa İman >Priority: Major > Attachments: TEZ-4192.1.patch, TEZ-4192.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > There is no way to pass information from scheduler to communicator. Sometimes > we want to let communicator know about extra properties of the task other > than its spec. One example is if a task is preemptable. > When scheduler receives a task, its taskspec is already constructed. It is a > nice immutable object at this point. We do not want to modify and put more > info in it at that stage. So we pass a new piece of information to > TaskSchedulerContext#taskAllocated along with the allocation. This info is > passed to the communicator untouched. It is up to the communicator > implementation to make use of it. Communicator implementation needs to > override a new registerRunningTaskAttempt that accepts this additional object > along with all the other arguments in the original registerRunningTaskAttempt > method. > The new registerRunningTaskAttempt method has a default implementation that > falls back to the old method. So if the user does not override the new > method, the same old behavior is preserved. Similarly on scheduler side, the > user needs to call new taskAllocated method to pass the additional info > object. If they use the old taskAllocated method, old behavior is preserved. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4272) Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED org.apache.hadoop.service.ServiceStateException: java.lang.ExceptionInInitializerError
[ https://issues.apache.org/jira/browse/TEZ-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331060#comment-17331060 ] Jonathan Turner Eagles commented on TEZ-4272: - Currently, there is not support for hadoop 3.3. Please use TEZ-4150 to track > Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED > org.apache.hadoop.service.ServiceStateException: > java.lang.ExceptionInInitializerError > -- > > Key: TEZ-4272 > URL: https://issues.apache.org/jira/browse/TEZ-4272 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.2, 0.10.0 >Reporter: Nomen'Aina >Priority: Major > Attachments: log.txt > > > Hello to all. > Currently I am trying to install Apache tez by following step by step, the > instructions on the official website, and when I run the command: > $HADOOP_PREFIX/bin/hadoop jar tez-examples.jar orderedwordcount > /tmp/input/file.txt /tmp/outout > I get the following errors: > Service org.apache.tez.dag.app.rm.TaskSchedulerManager failed in state > STARTED java.lang.NullPointerException at > org.apache.tez.dag.app.rm.TaskSchedulerManager.serviceStart(TaskSchedulerManager.java:653) > and > Error starting DAGAppMaster org.apache.hadoop.service.ServiceStateException: > java.lang.ExceptionInInitializerError > > > hadoop version:3.3.0 / tez version: 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TEZ-1019) Re-factor routing of events to use common code path for normal and recovery flow.
[ https://issues.apache.org/jira/browse/TEZ-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331048#comment-17331048 ] Jonathan Turner Eagles edited comment on TEZ-1019 at 4/23/21, 9:19 PM: --- Closing recovery v1 bugs/features. was (Author: jeagles): VertexEventSourceVertexRecovered > Re-factor routing of events to use common code path for normal and recovery > flow. > - > > Key: TEZ-1019 > URL: https://issues.apache.org/jira/browse/TEZ-1019 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-1019-2.patch, TEZ-1019-3.patch, TEZ-1019-4.patch, > TEZ-1019-5.patch, Tez-1019.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-15) Support for DAG AM recovery
[ https://issues.apache.org/jira/browse/TEZ-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-15. --- Resolution: Fixed > Support for DAG AM recovery > --- > > Key: TEZ-15 > URL: https://issues.apache.org/jira/browse/TEZ-15 > Project: Apache Tez > Issue Type: Improvement >Reporter: Bikas Saha >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-2544) Incorrect dag result due to wrong TaskSpec in recovering
[
https://issues.apache.org/jira/browse/TEZ-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-2544.
-
Resolution: Won't Fix
Closing recovery v1 bugs/features.
> Incorrect dag result due to wrong TaskSpec in recovering
>
>
> Key: TEZ-2544
> URL: https://issues.apache.org/jira/browse/TEZ-2544
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Critical
> Labels: Recovery
>
> Expected TaskSpec
> {noformat}
> DAGName : OrderedWordCount, VertexName: Summation, VertexParallelism: 1,
> TaskAttemptID:attempt_1433850314856_0019_1_01_00_0,
> processorName=org.apache.tez.examples.OrderedWordCount$SumProcessor,
> inputSpecListSize=1,
> outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=Tokenizer,
> physicalEdgeCount=2,
> inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }},
> ], outputSpecList=[{{ destinationVertexName=Sorter, physicalEdgeCount=1,
> outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput
> }}
> {noformat}
> The actual TaskSpec
> {noformat}
> DAGName : OrderedWordCount, VertexName: Summation, VertexParallelism: 1,
> TaskAttemptID:attempt_1433850314856_0019_1_01_00_0,
> processorName=org.apache.tez.examples.OrderedWordCount$SumProcessor,
> inputSpecListSize=1,
> outputSpecListSize=1, inputSpecList=[{{ sourceVertexName=Tokenizer,
> physicalEdgeCount=1,
> inputClassName=org.apache.tez.runtime.library.input.OrderedGroupedKVInput }},
> ], outputSpecList=[{{ destinationVertexName=Sorter, physicalEdgeCount=1,
> outputClassName=org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput
> }}
> {noformat}
> The expected physicalEdgeCount is 2 but actually it is 1, it happens when
> dynamic parallelism estimation is enabled.
> The cause is that Task is recovering but its vertex's source edge manager has
> not been updated from ScatterGatherEdgeManager to CustomShuffleEdgeManager,
> so will result in different physicalEdgeCount for InputSpec
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TEZ-2477) Session stats should be recovered
[
https://issues.apache.org/jira/browse/TEZ-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-2477.
-
Resolution: Won't Fix
Closing recovery v1 bugs/features.
> Session stats should be recovered
> --
>
> Key: TEZ-2477
> URL: https://issues.apache.org/jira/browse/TEZ-2477
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Major
>
> {code}
> diagResult.add("Session stats:"
> + "submittedDAGs=" + submittedDAGs.get()
> + ", successfulDAGs=" + successfulDAGs.get()
> + ", failedDAGs=" + failedDAGs.get()
> + ", killedDAGs=" + killedDAGs.get());
> {code}
> These data can be recovered from recovery summary file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TEZ-2456) Refactor recovery event logging to ensure it meet the recovery event spec
[ https://issues.apache.org/jira/browse/TEZ-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-2456. - Resolution: Won't Fix Closing recovery v1 bugs/features. > Refactor recovery event logging to ensure it meet the recovery event spec > - > > Key: TEZ-2456 > URL: https://issues.apache.org/jira/browse/TEZ-2456 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Labels: Recovery > > Currently we don't have spec for the recovery event logging. Recovery would > be fragile to code change. This jira try to define the spec and refactor the > recovery event logging to ensure it meet the spec. [~hitesh] Please help > review the following spec I drafted. > *DAG* > * DAGSubmitted/DAGInitializedEvent/DAGStartedEvent must been logged once, > Should not log it again when it’s recovered. > * DAGFinishedEvent may be logged multiple times. ( DAG move from SUCCEEDED > from ERROR ? Should we ignore this ? ) > * VertexFinishedEvent should be logged before DAGFinishedEvent > *Vertex* > * RootInputDataInformation must be logged before VertexInitializedEvent > * DataMovement must be logged before TaskFinishedEvent > * TaskFinishedEvent must be logged before VertexFinishedEvent > * VertexInitializedEvent / VertexStartedEvent should only be logged once, > should not log again when it’s recovered. > * VertexFinishedEvent may be logged multiple times. (e.g. Vertex move from > SUCCEEDED to FAILED) > * VertexParallelismUpdatedEvent must be logged before TaskStartedEvent > * For VertexFinishedEvent (SUCCEEDED), before it there must be at least n > TaskFinishedEvent (SUCCEEDED) > *Task* > * If there’s no TaskStartedEvent, TaskFinishedEvent may still be logged (e.g. > Task is killed in NEW ) Current’s behavior is that TaskFinishedEvent won’t > be logged if there’s no TaskStartedEvent. > * TaskStartedEvent should only be logged once. Should not log again when > it’s recovered. > * TaskFinishedEvent may be logged multiple times (e.g. Task move from > SUCCEEDED to FAILED) > * For TaskFinishedEvent (SUCCEEDED), before it there must be at least one > TaskAttemptFinishedEvent (SUCCEEDED) > > *TaskAttempt* > * If there’s no TaskAttemptStartedEvent, TaskAttemptFinishedEvent may still > be logged ( e.g. TaskAttempt is killed in NEW ) Current’s behavior is that > TaskAttemptFinishedEvent won’t be logged if there’s no TaskAttemptStartedEvent > * TaskAttemptStartedEvent should only be logged once. Should not log again > when it’s recovered. > * TaskAttemptFinishedEvent may be logged multiple times. (e.g. TaskAttempt > move from SUCCEEDED to FAILED) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-2418) TASK_ATTEMPT_FAILED_EVENT and TASK_COMPLETED_EVENT should move back to direct routing to attempt
[ https://issues.apache.org/jira/browse/TEZ-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-2418. - Resolution: Won't Fix Closing recovery v1 bugs/features. > TASK_ATTEMPT_FAILED_EVENT and TASK_COMPLETED_EVENT should move back to direct > routing to attempt > > > Key: TEZ-2418 > URL: https://issues.apache.org/jira/browse/TEZ-2418 > Project: Apache Tez > Issue Type: Bug >Reporter: Bikas Saha >Priority: Major > Attachments: TEZ-2418.1.patch > > > Due to recovery code path, they are currently double routed to the vertex > first and then the attempt. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-2431) Recovery of task events (eg. datamovement events) should not depend on ordering of task attempt events
[ https://issues.apache.org/jira/browse/TEZ-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-2431. - Resolution: Won't Fix Closing recovery v1 bugs/features. > Recovery of task events (eg. datamovement events) should not depend on > ordering of task attempt events > -- > > Key: TEZ-2431 > URL: https://issues.apache.org/jira/browse/TEZ-2431 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Priority: Major > > Today, task attempt events need to go through verteximpl before reaching the > task in order to maintain ordering guarantees for recovery. This causes these > events to be routed twice through the dispatcher. This can cause overhead > delays in large jobs. Also, this makes assumptions about event ordering which > make the system fragile. Recovery should work independently of other system > interactions so that evolution of other components is not affected by > recovery unless it affects recovery logically. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-2107) Recovery failure in the case of Auto-reduce parallelism
[
https://issues.apache.org/jira/browse/TEZ-2107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-2107.
-
Resolution: Won't Fix
Closing recovery v1 bugs/features.
> Recovery failure in the case of Auto-reduce parallelism
> ---
>
> Key: TEZ-2107
> URL: https://issues.apache.org/jira/browse/TEZ-2107
> Project: Apache Tez
> Issue Type: Sub-task
>Affects Versions: 0.6.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Major
>
> The following errors happens when recovering in the case of auto-reduce
> parallelism. The task number is reduced from 2 to 1. while the upstream
> vertex's DataMovementEvent is still routed to task 2 which has been removed
> when auto-reduce parallelism.
> {code}
> 2015-02-16 09:11:54,587 FATAL [Dispatcher thread: Central]
> common.AsyncDispatcher: Error in dispatcher thread
> org.apache.tez.dag.api.TezUncheckedException: Unexpected null task.
> sourceVertex=vertex_1424048826974_0002_1_00 [scope-47] srcTaskIndex = 0
> destVertex=vertex_1424048826974_0002_1_01 [scope-50] destTaskIndex=1
> destNumTasks=1
> edgeManager=org.apache.tez.dag.app.dag.impl.ScatterGatherEdgeManager
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendDmEventOrIfEventToTasks(Edge.java:358)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:422)
> at
> org.apache.tez.dag.app.dag.impl.Edge.handleCompositeDataMovementEvent(Edge.java:310)
> at
> org.apache.tez.dag.app.dag.impl.Edge.sendTezEventToDestinationTasks(Edge.java:378)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.handleRoutedTezEvents(VertexImpl.java:3795)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl.access$3600(VertexImpl.java:187)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:3708)
> at
> org.apache.tez.dag.app.dag.impl.VertexImpl$RouteEventTransition.transition(VertexImpl.java:3700)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
> at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1575)
> at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:186)
> at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1802)
> at
> org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1788)
> at
> org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The following exception will also happen sometimes
> {code}
> 2015-06-10 08:02:03,417 ERROR [Dispatcher thread: Central] impl.VertexImpl:
> Exception in VertexManager, vertex:vertex_1433894507873_0001_1_01 [Summation]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> org.apache.tez.dag.api.TezUncheckedException: Atleast 1 bipartite source
> should exist, vertexName=Summation
> at
> org.apache.tez.dag.app.dag.impl.VertexManager$VertexManagerCallback.onFailure(VertexManager.java:516)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:977)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
> at
> com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
> at
> com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:86)
> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380)
> at java.util.concurrent.FutureTask.setException(FutureTask.java:247)
> at java.util.concurrent.FutureTask.run(FutureTask.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.tez.dag.api.TezUncheckedException: Atleast 1 bipartite
> source should exist, vertexName=Summation
> at
>
[jira] [Resolved] (TEZ-1565) Add tests for checking custom vertex managers like auto-reduce parallelism in recovery
[ https://issues.apache.org/jira/browse/TEZ-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-1565. - Resolution: Won't Fix Closing recovery v1 bugs/features. > Add tests for checking custom vertex managers like auto-reduce parallelism in > recovery > -- > > Key: TEZ-1565 > URL: https://issues.apache.org/jira/browse/TEZ-1565 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-1384) Move recovery related code into inner class
[ https://issues.apache.org/jira/browse/TEZ-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-1384. - Resolution: Won't Fix Closing recovery v1 bugs/features. > Move recovery related code into inner class > --- > > Key: TEZ-1384 > URL: https://issues.apache.org/jira/browse/TEZ-1384 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Attachments: Tez-1384-2.patch, Tez-1384.patch > > > Currently each entity (DAG, Vertex, Task, TaskAttempt) has some common > recovery code like log history event and restore from history event. These > are 2 opposite aspects of recovery. One for store status while the other is > for restore status. This jira is for putting these pieces of code together ( > in an inner class ). In this way, it is easy to maintain and cut down the > possibility that one field is not stored or restored. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-1019) Re-factor routing of events to use common code path for normal and recovery flow.
[ https://issues.apache.org/jira/browse/TEZ-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-1019. - Resolution: Won't Fix VertexEventSourceVertexRecovered > Re-factor routing of events to use common code path for normal and recovery > flow. > - > > Key: TEZ-1019 > URL: https://issues.apache.org/jira/browse/TEZ-1019 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-1019-2.patch, TEZ-1019-3.patch, TEZ-1019-4.patch, > TEZ-1019-5.patch, Tez-1019.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-1156) setParallelism in recovery does not send event to downstream vertices
[ https://issues.apache.org/jira/browse/TEZ-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-1156. - Resolution: Won't Fix Closing recovery v1 bugs/features. > setParallelism in recovery does not send event to downstream vertices > - > > Key: TEZ-1156 > URL: https://issues.apache.org/jira/browse/TEZ-1156 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Priority: Major > > In non-recovery case, downstream vertices are notified. If say, parallelism > is update and saved but the AM dies before downstream vertices get the event > and update their parallelism then the recovered AM will set the parallelism > of the recovered vertex but the downstream vertices may remain uninitialized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-993) Remove application logic from RecoveryService
[ https://issues.apache.org/jira/browse/TEZ-993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-993. Resolution: Won't Fix Closing recovery v1 bugs/features. > Remove application logic from RecoveryService > - > > Key: TEZ-993 > URL: https://issues.apache.org/jira/browse/TEZ-993 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-993-3.patch, TEZ-993-4.patch, TEZ-993-5.patch, > Tez-993-2.patch, Tez-993.patch > > > Currently RecoveryService storage logic knows a lot about the DAG like which > dag is pre-warm and does not need to be stored, which events needs special > treatment etc. This kind of logic couples the DAG and the storage more than > is probably necessary and can be a source of complications down the road. The > storage should ideally be simply storing a sequence of arbitrary records > delimited by a marker. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-861) Handle task re-schedules in recovery
[ https://issues.apache.org/jira/browse/TEZ-861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-861. Resolution: Won't Fix Closing recovery v1 bugs/features. > Handle task re-schedules in recovery > > > Key: TEZ-861 > URL: https://issues.apache.org/jira/browse/TEZ-861 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-992) Recovery data should not be written on AsyncDispatcher thread
[ https://issues.apache.org/jira/browse/TEZ-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-992. Resolution: Won't Fix Closing recovery v1 bugs/features. > Recovery data should not be written on AsyncDispatcher thread > - > > Key: TEZ-992 > URL: https://issues.apache.org/jira/browse/TEZ-992 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Jeff Zhang >Priority: Major > Attachments: DAG_FinishSaving.gv, DAG_FinishSaving_2.gv, > DAG_FinishSaving_2.pdf, TEZ-992.patch, Vertex_FinishSaving.gv, > Vertex_FinishSaving_2.gv, Vertex_FinishSaving_2.pdf > > > This may block the DAG operations in case the recovery data needs to be > synchronously stored. The operations requiring this blocking operation should > change their state machines to wait for the store operation before moving > ahead. They will move ahead after they receive notification from the > RecoveryService that their operation has completed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-860) Handle restore of AMContainer and AMNode states on recovery
[ https://issues.apache.org/jira/browse/TEZ-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-860. Resolution: Won't Fix Closing recovery v1 bugs/features. > Handle restore of AMContainer and AMNode states on recovery > --- > > Key: TEZ-860 > URL: https://issues.apache.org/jira/browse/TEZ-860 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Priority: Major > > Required to handle retroactive failures or nodes going unhealthy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-852) Implement more optimal flush/sync mechanism to HDFS
[ https://issues.apache.org/jira/browse/TEZ-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-852. Resolution: Won't Fix Closing recovery v1 bugs/features. > Implement more optimal flush/sync mechanism to HDFS > --- > > Key: TEZ-852 > URL: https://issues.apache.org/jira/browse/TEZ-852 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-3421) cdh 5.5.2 cann't use tez 0.8.4
[
https://issues.apache.org/jira/browse/TEZ-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331035#comment-17331035
]
Jonathan Turner Eagles commented on TEZ-3421:
-
[~ft20082], Is this jira still relevant? Trying to close out old jiras.
> cdh 5.5.2 cann't use tez 0.8.4
> --
>
> Key: TEZ-3421
> URL: https://issues.apache.org/jira/browse/TEZ-3421
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.1, 0.8.4
> Environment: cdh 5.5.2, hive 1.1.0, tez 0.8.4
>Reporter: Jake.zhang
>Priority: Major
>
> following the install guide, and it don't work.
> {code}
> 2016-08-31 15:31:20,614 [INFO] [IPC Server handler 0 on 43797]
> |app.DAGAppMaster|: Running DAG:
> root_20160831153131_0c02cc1a-1447-4ce9-a4b1-f66c2b8e1354:2
> 2016-08-31 15:31:20,849 [INFO] [IPC Server handler 0 on 43797]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_SUBMITTED]:
> dagID=dag_1470895429219_0037_1, submitTime=1472628680567
> 2016-08-31 15:31:20,886 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Default container context for
> vertex_1470895429219_0037_1_00 [Map 1]=LocalResources: [[ name=tezlib,
> value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020
> file: "/user/tez/tez-0.8.4.tar.gz" } size: 48523006 timestamp: 1472625394859
> type: ARCHIVE visibility: PUBLIC],[ name=hbase-client.jar, value=resource {
> scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-client.jar"
> } size: 1294765 timestamp: 1472625755599 type: FILE visibility: PRIVATE],[
> name=hbase-server.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-server.jar"
> } size: 4299872 timestamp: 1472625755903 type: FILE visibility: PRIVATE],[
> name=hive-hbase-handler-1.1.0-cdh5.8.0.jar, value=resource { scheme: "hdfs"
> host: "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hive-hbase-handler-1.1.0-cdh5.8.0.jar"
> } size: 118398 timestamp: 1472625755443 type: FILE visibility: PRIVATE],[
> name=postgresql-9.0-801.jdbc4.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/postgresql-9.0-801.jdbc4.jar"
> } size: 539705 timestamp: 1472625756348 type: FILE visibility: PRIVATE],[
> name=hbase-common.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-common.jar"
> } size: 581623 timestamp: 1472625755634 type: FILE visibility: PRIVATE],[
> name=htrace-core-3.2.0-incubating.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core-3.2.0-incubating.jar"
> } size: 1483913 timestamp: 1472625756002 type: FILE visibility: PRIVATE],[
> name=hbase-hadoop-compat.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop-compat.jar"
> } size: 39226 timestamp: 1472625755671 type: FILE visibility: PRIVATE],[
> name=htrace-core4-4.0.1-incubating.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core4-4.0.1-incubating.jar"
> } size: 1485102 timestamp: 1472625756117 type: FILE visibility: PRIVATE],[
> name=hbase-protocol.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-protocol.jar"
> } size: 4378452 timestamp: 1472625755802 type: FILE visibility: PRIVATE],[
> name=htrace-core.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core.jar"
> } size: 1483913 timestamp: 1472625756052 type: FILE visibility: PRIVATE],[
> name=hbase-hadoop2-compat.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop2-compat.jar"
> } size: 97555 timestamp: 1472625755713 type: FILE visibility: PRIVATE],[
> name=tez-conf.pb, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
>
[jira] [Resolved] (TEZ-1522) Scheduling can result in out of order execution and slowdown of upstream work
[ https://issues.apache.org/jira/browse/TEZ-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-1522. - Resolution: Later DagAwareYarnTaskScheduler addresses a large portion of this jira. We can close this jira and have customers configure that scheduler. > Scheduling can result in out of order execution and slowdown of upstream work > - > > Key: TEZ-1522 > URL: https://issues.apache.org/jira/browse/TEZ-1522 > Project: Apache Tez > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Critical > Labels: performance > Attachments: TEZ-1522.1.wip.txt, TEZ-1522.2.wip.txt, > TEZ-1522.am.log.gz, task_runtime.svg > > > M2 M7 > \ / > (sg) \/ >R3/ (b) > \ / > (b) \ / > \ / > M5 > | > R6 > Plz refer to the attachment (task runtime SVG). In this case, M5 got > scheduled much earlier than R3 (green color in the diagram) and retained lots > of containers. > R3 got less containers to work with. > Attaching the output from the status monitor when the job ran; Map_5 has > taken up almost all of cluster resource, whereas Reducer_3 got fraction of > the capacity. > Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 > Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 > Reducer_6: 0/1 > > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 14(+7)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 63(+14)/8000 Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 159(+22)/8000Reducer_6: 0/1 > Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: > 308(+29)/8000Reducer_6: 0/1 > ... > Creating this JIRA as a placeholder for scheduler enhancement. One > possibililty could be to > schedule lesser number of tasks in downstream vertices, based on the > information available for the upstream vertex. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-3885) Tez timeline server integration error
[
https://issues.apache.org/jira/browse/TEZ-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles resolved TEZ-3885.
-
Resolution: Duplicate
Some portion of this jira was fixed in another jira.
> Tez timeline server integration error
> -
>
> Key: TEZ-3885
> URL: https://issues.apache.org/jira/browse/TEZ-3885
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Affects Versions: 0.9.0
> Environment: apache hive version - 3.0.0-SNAPSHOT
> apache hadoop version - 3.0.0
> tez version - 0.9.0
>Reporter: Yousun Jeong
>Priority: Critical
> Labels: beginner
>
> If I configure below, I've got an error about launch hive error. I think it's
> a library problem, but I'm not sure how to fix it.
>
>
> tez.lib.uris
> hdfs:///apps/tez-0.9.0/tez-0.9.0-minimal.tar.gz
>
>
> tez.use.cluster.hadoop-libs
> true
>
>
> Enable Tez to use the Timeline Server for History
> Logging
> tez.history.logging.service.class
>
> org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService
>
>
> URL for where the Tez UI is hosted
> tez.tez-ui.history-url.base
> http://HOST:8081/tez-ui-0.9.0/
>
>
> 2018-01-09 17:47:55,484 [INFO] [main] |app.DAGAppMaster|: Creating
> DAGAppMaster for applicationId=application_1515483495537_0016, attemptNum=1,
> AMContainerId=container_e40_1515483495537_0016_01_01, jvmPid=127852,
> userFromEnv=hadoop, cliSessionOption=true,
> pwd=/data5/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016/container_e40_1515483495537_0016_01_01,
>
> localDirs=/data1/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016,/data2/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016,/data3/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016,/data4/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016,/data5/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016,/data6/yarn/local/usercache/hadoop/appcache/application_1515483495537_0016,
>
> logDirs=/data1/yarn/logs/application_1515483495537_0016/container_e40_1515483495537_0016_01_01
> 2018-01-09 17:47:55,588 [INFO] [main] |Configuration.deprecation|:
> yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead,
> use yarn.system-metrics-publisher.enabled
> 2018-01-09 17:47:55,727 [INFO] [main] |security.SecurityUtil|: Updating
> Configuration
> 2018-01-09 17:47:55,867 [INFO] [main] |app.DAGAppMaster|: Created
> DAGAppMaster for application appattempt_1515483495537_0016_01,
> versionInfo=[ component=tez-dag, version=0.9.0, revision=${buildNumber},
> SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
> buildTime=2018-01-09T00:23:13Z ]
> 2018-01-09 17:47:55,897 [INFO] [main] |app.DAGAppMaster|:
> /
> [system properties]
> os.name: Linux
> os.version: 2.6.32-573.7.1.el6.x86_64
> java.home: /usr/java/jdk1.8.0_121/jre
> java.runtime.version: 1.8.0_121-b13
> java.vendor: Oracle Corporation
> java.version: 1.8.0_121
> java.vm.name: Java HotSpot(TM) 64-Bit Server VM
> java.class.path:
>
[jira] [Resolved] (TEZ-3273) app.TaskAttemptListenerImpTezDag: Attempt is not recognized for heartbeat in tez 0.5.2,cause job hang
[ https://issues.apache.org/jira/browse/TEZ-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-3273. - Resolution: Incomplete A number of hangs have been fixed in 0.7/0.9/0.10 releases as well as a number of scheduler changes including a new DagAwareYarnTaskScheduler that has been stable now for a few years. Closing this jira as it has grown stale. > app.TaskAttemptListenerImpTezDag: Attempt is not recognized for heartbeat in > tez 0.5.2,cause job hang > - > > Key: TEZ-3273 > URL: https://issues.apache.org/jira/browse/TEZ-3273 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.5.2 > Environment: hive0.14 hadoop2.6 >Reporter: Feng Yuan >Priority: Critical > Attachments: app_logs.zip > > > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > Map 1: 145(+0,-1)/146 Reducer 2: 0/415 > stuck forever~ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4157) ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez
[ https://issues.apache.org/jira/browse/TEZ-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330837#comment-17330837 ] Jonathan Turner Eagles commented on TEZ-4157: - Sorry. The comment didn't show up for me for some reason. [~abstractdog], I worried the tez shuffle handler auxservices/ShuffleHandler.java will NOT work for branch-0.9 as nodemanager api is netty 3 in hadoop 2.10/2.8. Have you verified this branch-0.9 shuffle handler works on 2.10 with this change? > ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez > --- > > Key: TEZ-4157 > URL: https://issues.apache.org/jira/browse/TEZ-4157 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.1, 0.9.3 > > Attachments: TEZ-4157.01.patch, TEZ-4157.02.patch, TEZ-4157.03.patch, > TEZ-4157.04.patch, TEZ-4157.05.patch, TEZ-4157.06.patch, dep.log > > Time Spent: 1h > Remaining Estimate: 0h > > -In the dependency tree, there are 2 occurrences of compile scope direct > netty dependencies, however, they're not used at all. I compiled locally > successfully without them. E.g. when investigating blackduck alerts > (complaining about netty deps for current 3.10.5.Final), it would be cleaner > to start from a dependency tree where Tez doesn't depend on netty directly in > order to eliminate its responsibility (and move the focus to underlying > hadoop for instance).- > Tez depends on netty3 almost only in ShuffleHandler and some related classes. > We can eliminate netty3 by upgrading it, but this effort might involve some > testing due to fundamental [changes from > netty3->netty4|https://netty.io/wiki/new-and-noteworthy-in-4.0.html] + we > don't have a reference yet, as [hadoop's > ShuffleHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java] > is still on netty3. > As per the netty documentation, we can also expect some performance > improvement (e.g. Pooled buffers). > Background: > netty4 migration guideline: > https://netty.io/wiki/new-and-noteworthy-in-4.0.html > articles of possible performance improvement: > https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html > https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/ > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4157) ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez
[ https://issues.apache.org/jira/browse/TEZ-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17330833#comment-17330833 ] Jonathan Turner Eagles commented on TEZ-4157: - [~abstractdog], I don't see a review on this code before it was merged. Did I miss the review? > ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez > --- > > Key: TEZ-4157 > URL: https://issues.apache.org/jira/browse/TEZ-4157 > Project: Apache Tez > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.1, 0.9.3 > > Attachments: TEZ-4157.01.patch, TEZ-4157.02.patch, TEZ-4157.03.patch, > TEZ-4157.04.patch, TEZ-4157.05.patch, TEZ-4157.06.patch, dep.log > > Time Spent: 1h > Remaining Estimate: 0h > > -In the dependency tree, there are 2 occurrences of compile scope direct > netty dependencies, however, they're not used at all. I compiled locally > successfully without them. E.g. when investigating blackduck alerts > (complaining about netty deps for current 3.10.5.Final), it would be cleaner > to start from a dependency tree where Tez doesn't depend on netty directly in > order to eliminate its responsibility (and move the focus to underlying > hadoop for instance).- > Tez depends on netty3 almost only in ShuffleHandler and some related classes. > We can eliminate netty3 by upgrading it, but this effort might involve some > testing due to fundamental [changes from > netty3->netty4|https://netty.io/wiki/new-and-noteworthy-in-4.0.html] + we > don't have a reference yet, as [hadoop's > ShuffleHandler|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java] > is still on netty3. > As per the netty documentation, we can also expect some performance > improvement (e.g. Pooled buffers). > Background: > netty4 migration guideline: > https://netty.io/wiki/new-and-noteworthy-in-4.0.html > articles of possible performance improvement: > https://blog.twitter.com/engineering/en_us/a/2013/netty-4-at-twitter-reduced-gc-overhead.html > https://developer.squareup.com/blog/upgrading-a-reverse-proxy-from-netty-3-to-4/ > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-861) Handle task re-schedules in recovery
[ https://issues.apache.org/jira/browse/TEZ-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325270#comment-17325270 ] Jonathan Turner Eagles commented on TEZ-861: This is already supported and later I will clean this and associated jiras up. [~balakuntala], are you looking for a way to contribute to this project? In that case, I could point you issues that may relate to how you would want to contribute. > Handle task re-schedules in recovery > > > Key: TEZ-861 > URL: https://issues.apache.org/jira/browse/TEZ-861 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Hitesh Shah >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4283) Docker fails to build on master - upgrade to nodejs10
[
https://issues.apache.org/jira/browse/TEZ-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17318012#comment-17318012
]
Jonathan Turner Eagles commented on TEZ-4283:
-
+1. This commit I believe has already gone in. [~abstractdog], Can we update
the status of this jira to reflect current status?
> Docker fails to build on master - upgrade to nodejs10
> -
>
> Key: TEZ-4283
> URL: https://issues.apache.org/jira/browse/TEZ-4283
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Blocker
> Fix For: 0.10.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/blue/organizations/jenkins/tez-multibranch/detail/PR-102/5/pipeline
> {code}
> [2021-02-08T10:48:03.916Z] WARN engine npm@7.5.2: wanted: {"node":">=10"}
> (current: {"node":"8.10.0","npm":"3.5.2"})
> [2021-02-08T10:48:03.916Z] WARN engine npm@7.5.2: wanted: {"node":">=10"}
> (current: {"node":"8.10.0","npm":"3.5.2"})
> [2021-02-08T10:48:14.281Z] /usr/local/lib
> [2021-02-08T10:48:14.281Z] `-- (empty)
> [2021-02-08T10:48:14.281Z]
> [2021-02-08T10:48:14.281Z] npm ERR! Linux 4.15.0-112-generic
> [2021-02-08T10:48:14.281Z] npm ERR! argv "/usr/bin/node" "/usr/bin/npm"
> "install" "npm@latest" "-g"
> [2021-02-08T10:48:14.281Z] npm ERR! node v8.10.0
> [2021-02-08T10:48:14.281Z] npm ERR! npm v3.5.2
> [2021-02-08T10:48:14.281Z] npm ERR! path
> /usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552
> [2021-02-08T10:48:14.281Z] npm ERR! code ENOENT
> [2021-02-08T10:48:14.281Z] npm ERR! errno -2
> [2021-02-08T10:48:14.281Z] npm ERR! syscall rename
> [2021-02-08T10:48:14.281Z]
> [2021-02-08T10:48:14.281Z] npm ERR! enoent ENOENT: no such file or directory,
> rename '/usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552' ->
> '/usr/local/lib/node_modules/npm/node_modules/@npmcli/ci-detect'
> [2021-02-08T10:48:14.281Z] npm ERR! enoent ENOENT: no such file or directory,
> rename '/usr/local/lib/node_modules/.staging/@npmcli/ci-detect-c7bf9552' ->
> '/usr/local/lib/node_modules/npm/node_modules/@npmcli/ci-detect'
> [2021-02-08T10:48:14.281Z] npm ERR! enoent This is most likely not a problem
> with npm itself
> [2021-02-08T10:48:14.281Z] npm ERR! enoent and is related to npm not being
> able to find a file.
> [2021-02-08T10:48:14.281Z] npm ERR! enoent
> [2021-02-08T10:48:14.281Z]
> [2021-02-08T10:48:14.281Z] npm ERR! Please include the following file with
> any support request:
> [2021-02-08T10:48:14.282Z] npm ERR! /root/npm-debug.log
> [2021-02-08T10:48:14.282Z] npm ERR! code 1
> [2021-02-08T10:48:14.806Z] The command '/bin/bash -o pipefail -c apt-get -q
> update && apt-get install -y --no-install-recommends nodejs npm &&
> apt-get clean && rm -rf /var/lib/apt/lists/* && npm install
> npm@latest -g && npm install -g jshint' returned a non-zero code: 1
> [2021-02-08T10:48:14.806Z] ERROR: Docker failed to build yetus/tez:474bb22ba.
> {code}
> https://github.com/npm/ci-detect
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TEZ-4299) Default java opts cause jdk11 to fail
[
https://issues.apache.org/jira/browse/TEZ-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated TEZ-4299:
Parent: TEZ-4020
Issue Type: Sub-task (was: Bug)
> Default java opts cause jdk11 to fail
> -
>
> Key: TEZ-4299
> URL: https://issues.apache.org/jira/browse/TEZ-4299
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
>
> From TEZ-4298 PR request on jdk11
> {noformat}
> [INFO] Running org.apache.tez.auxservices.TestShuffleHandlerJobs
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 51.727 s <<< FAILURE! - in org.apache.tez.auxservices.TestShuffleHandlerJobs
> [ERROR]
> org.apache.tez.auxservices.TestShuffleHandlerJobs.testOrderedWordCount Time
> elapsed: 4.583 s <<< ERROR!
> org.apache.tez.dag.api.SessionNotRunning:
> TezSession has already shutdown. Application application_1616693699949_0001
> failed 2 times due to AM Container for appattempt_1616693699949_0001_02
> exited with exitCode: 1
> Failing this attempt.Diagnostics: [2021-03-25 17:35:06.864]Exception from
> container-launch.
> Container id: container_1616693699949_0001_02_01
> Exit code: 1
> [2021-03-25 17:35:06.874]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Unrecognized VM option 'PrintGCTimeStamps'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> [2021-03-25 17:35:06.876]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Unrecognized VM option 'PrintGCTimeStamps'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> For more detailed output, check the application tracking page:
> http://a5bb2fa1dfc2:34397/cluster/app/application_1616693699949_0001 Then
> click on links to logs of each attempt.
> . Failing the application.
> at
> org.apache.tez.auxservices.TestShuffleHandlerJobs.testOrderedWordCount(TestShuffleHandlerJobs.java:129)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TEZ-4299) Default java opts cause jdk11 to fail
[
https://issues.apache.org/jira/browse/TEZ-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated TEZ-4299:
Description:
>From TEZ-4298 PR request on jdk11
{noformat}
[INFO] Running org.apache.tez.auxservices.TestShuffleHandlerJobs
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 51.727
s <<< FAILURE! - in org.apache.tez.auxservices.TestShuffleHandlerJobs
[ERROR] org.apache.tez.auxservices.TestShuffleHandlerJobs.testOrderedWordCount
Time elapsed: 4.583 s <<< ERROR!
org.apache.tez.dag.api.SessionNotRunning:
TezSession has already shutdown. Application application_1616693699949_0001
failed 2 times due to AM Container for appattempt_1616693699949_0001_02
exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-03-25 17:35:06.864]Exception from
container-launch.
Container id: container_1616693699949_0001_02_01
Exit code: 1
[2021-03-25 17:35:06.874]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Unrecognized VM option 'PrintGCTimeStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[2021-03-25 17:35:06.876]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Unrecognized VM option 'PrintGCTimeStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
For more detailed output, check the application tracking page:
http://a5bb2fa1dfc2:34397/cluster/app/application_1616693699949_0001 Then click
on links to logs of each attempt.
. Failing the application.
at
org.apache.tez.auxservices.TestShuffleHandlerJobs.testOrderedWordCount(TestShuffleHandlerJobs.java:129)
{noformat}
> Default java opts cause jdk11 to fail
> -
>
> Key: TEZ-4299
> URL: https://issues.apache.org/jira/browse/TEZ-4299
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
>
> From TEZ-4298 PR request on jdk11
> {noformat}
> [INFO] Running org.apache.tez.auxservices.TestShuffleHandlerJobs
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 51.727 s <<< FAILURE! - in org.apache.tez.auxservices.TestShuffleHandlerJobs
> [ERROR]
> org.apache.tez.auxservices.TestShuffleHandlerJobs.testOrderedWordCount Time
> elapsed: 4.583 s <<< ERROR!
> org.apache.tez.dag.api.SessionNotRunning:
> TezSession has already shutdown. Application application_1616693699949_0001
> failed 2 times due to AM Container for appattempt_1616693699949_0001_02
> exited with exitCode: 1
> Failing this attempt.Diagnostics: [2021-03-25 17:35:06.864]Exception from
> container-launch.
> Container id: container_1616693699949_0001_02_01
> Exit code: 1
> [2021-03-25 17:35:06.874]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Unrecognized VM option 'PrintGCTimeStamps'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> [2021-03-25 17:35:06.876]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Unrecognized VM option 'PrintGCTimeStamps'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> For more detailed output, check the application tracking page:
> http://a5bb2fa1dfc2:34397/cluster/app/application_1616693699949_0001 Then
> click on links to logs of each attempt.
> . Failing the application.
> at
> org.apache.tez.auxservices.TestShuffleHandlerJobs.testOrderedWordCount(TestShuffleHandlerJobs.java:129)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TEZ-4299) Default java opts cause jdk11 to fail
Jonathan Turner Eagles created TEZ-4299: --- Summary: Default java opts cause jdk11 to fail Key: TEZ-4299 URL: https://issues.apache.org/jira/browse/TEZ-4299 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4298) ShuffleHandler is not source compatible with Hadoop 3.3.0
[
https://issues.apache.org/jira/browse/TEZ-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles updated TEZ-4298:
Attachment: TEZ-4298.001.patch
> ShuffleHandler is not source compatible with Hadoop 3.3.0
> -
>
> Key: TEZ-4298
> URL: https://issues.apache.org/jira/browse/TEZ-4298
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Turner Eagles
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Attachments: TEZ-4298.001.patch
>
>
> {noformat}
> mvn clean package -DskipTests -pl '!tez-ui' -Dhadoop.version=3.3.0
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[764,45]
> incompatible types: com.google.protobuf.ByteString cannot be converted to
> org.apache.hadoop.thirdparty.protobuf.ByteString
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TEZ-4298) ShuffleHandler is not source compatible with Hadoop 3.3.0
Jonathan Turner Eagles created TEZ-4298:
---
Summary: ShuffleHandler is not source compatible with Hadoop 3.3.0
Key: TEZ-4298
URL: https://issues.apache.org/jira/browse/TEZ-4298
Project: Apache Tez
Issue Type: Bug
Reporter: Jonathan Turner Eagles
Assignee: Jonathan Turner Eagles
{noformat}
mvn clean package -DskipTests -pl '!tez-ui' -Dhadoop.version=3.3.0
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[764,45]
incompatible types: com.google.protobuf.ByteString cannot be converted to
org.apache.hadoop.thirdparty.protobuf.ByteString
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (TEZ-4294) Tez UI counters missing in settings view
Jonathan Turner Eagles created TEZ-4294: --- Summary: Tez UI counters missing in settings view Key: TEZ-4294 URL: https://issues.apache.org/jira/browse/TEZ-4294 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Turner Eagles Assignee: Jonathan Turner Eagles org.apache.tez.common.counters.TaskCounter.INPUT_SPLIT_LENGTH_BYTES is one missing example Need to compare tez-api/src/main/java/org/apache/tez/common/counters/TaskCounter.java to tez-ui/src/main/webapp/config/default-app-conf.js to see if any others are missing and valuable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4289) Remove Dependency on commons-math
[ https://issues.apache.org/jira/browse/TEZ-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4289. - Fix Version/s: 0.9.3 0.10.1 Resolution: Fixed +1 Committed to master and branch-0.9 > Remove Dependency on commons-math > - > > Key: TEZ-4289 > URL: https://issues.apache.org/jira/browse/TEZ-4289 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 0.10.1, 0.9.3 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TEZ-4288) Upgrade SLF4J to 1.7.30
[ https://issues.apache.org/jira/browse/TEZ-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles updated TEZ-4288: Fix Version/s: 0.9.3 > Upgrade SLF4J to 1.7.30 > --- > > Key: TEZ-4288 > URL: https://issues.apache.org/jira/browse/TEZ-4288 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 0.10.1, 0.9.3 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4288) Upgrade SLF4J to 1.7.30
[ https://issues.apache.org/jira/browse/TEZ-4288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4288. - Fix Version/s: 0.10.1 Resolution: Fixed +1. Committed to master branch via github. I will cherry-pick this to branch-0.9 as well. > Upgrade SLF4J to 1.7.30 > --- > > Key: TEZ-4288 > URL: https://issues.apache.org/jira/browse/TEZ-4288 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 0.10.1 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4241) UI: containerLogs testWrapper test fails
[
https://issues.apache.org/jira/browse/TEZ-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283828#comment-17283828
]
Jonathan Turner Eagles commented on TEZ-4241:
-
I have assigned this jira to myself to indicate contributor of 02 patch. Pushed
to master and cherry-picked to branch-0.9.
Thanks to [~abstractdog] and [~euigeun_chung] for their help to get this work
committed.
> UI: containerLogs testWrapper test fails
>
>
> Key: TEZ-4241
> URL: https://issues.apache.org/jira/browse/TEZ-4241
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: László Bodor
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Fix For: 0.10.1
>
> Attachments: TEZ-4241.01.patch, TEZ-4241.02.patch, TEZ-4241.03.patch,
> patch-unit-root.txt
>
>
> This causes failures in precommit for patches that are independent of tez ui.
> {code}
> not ok 750 PhantomJS 2.1 - Unit | Serializer | dag: containerLogs test
> ---
> actual: >
> null
> message: >
> Died on test #2
> testWrapper@http://localhost:7357/assets/test-support.js:6240:16
> test@http://localhost:7357/assets/test-support.js:6254:44
> http://localhost:7357/assets/tests.js:19223:24
> http://localhost:7357/assets/vendor.js:477:34
> tryFinally@http://localhost:7357/assets/vendor.js:358:21
> requireModule@http://localhost:7357/assets/vendor.js:475:15
> require@http://localhost:7357/assets/test-loader.js:60:16
> loadModules@http://localhost:7357/assets/test-loader.js:51:25
> load@http://localhost:7357/assets/test-loader.js:82:35
> http://localhost:7357/assets/test-support.js:6037:20: undefined
> is not an object (evaluating 'this.get')
> Log: |
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4241) UI: containerLogs testWrapper test fails
[
https://issues.apache.org/jira/browse/TEZ-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283827#comment-17283827
]
Jonathan Turner Eagles commented on TEZ-4241:
-
[~euigeun_chung], we're going to commit work-around patch above 02.patch to get
the builds passing again. A proper fix in a follow-on jira. Please option a
jira if you wish to pursue that.
> UI: containerLogs testWrapper test fails
>
>
> Key: TEZ-4241
> URL: https://issues.apache.org/jira/browse/TEZ-4241
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: László Bodor
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Fix For: 0.10.1
>
> Attachments: TEZ-4241.01.patch, TEZ-4241.02.patch, TEZ-4241.03.patch,
> patch-unit-root.txt
>
>
> This causes failures in precommit for patches that are independent of tez ui.
> {code}
> not ok 750 PhantomJS 2.1 - Unit | Serializer | dag: containerLogs test
> ---
> actual: >
> null
> message: >
> Died on test #2
> testWrapper@http://localhost:7357/assets/test-support.js:6240:16
> test@http://localhost:7357/assets/test-support.js:6254:44
> http://localhost:7357/assets/tests.js:19223:24
> http://localhost:7357/assets/vendor.js:477:34
> tryFinally@http://localhost:7357/assets/vendor.js:358:21
> requireModule@http://localhost:7357/assets/vendor.js:475:15
> require@http://localhost:7357/assets/test-loader.js:60:16
> loadModules@http://localhost:7357/assets/test-loader.js:51:25
> load@http://localhost:7357/assets/test-loader.js:82:35
> http://localhost:7357/assets/test-support.js:6037:20: undefined
> is not an object (evaluating 'this.get')
> Log: |
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (TEZ-4241) UI: containerLogs testWrapper test fails
[
https://issues.apache.org/jira/browse/TEZ-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Turner Eagles reassigned TEZ-4241:
---
Assignee: Jonathan Turner Eagles (was: László Bodor)
> UI: containerLogs testWrapper test fails
>
>
> Key: TEZ-4241
> URL: https://issues.apache.org/jira/browse/TEZ-4241
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: László Bodor
>Assignee: Jonathan Turner Eagles
>Priority: Major
> Fix For: 0.10.1
>
> Attachments: TEZ-4241.01.patch, TEZ-4241.02.patch, TEZ-4241.03.patch,
> patch-unit-root.txt
>
>
> This causes failures in precommit for patches that are independent of tez ui.
> {code}
> not ok 750 PhantomJS 2.1 - Unit | Serializer | dag: containerLogs test
> ---
> actual: >
> null
> message: >
> Died on test #2
> testWrapper@http://localhost:7357/assets/test-support.js:6240:16
> test@http://localhost:7357/assets/test-support.js:6254:44
> http://localhost:7357/assets/tests.js:19223:24
> http://localhost:7357/assets/vendor.js:477:34
> tryFinally@http://localhost:7357/assets/vendor.js:358:21
> requireModule@http://localhost:7357/assets/vendor.js:475:15
> require@http://localhost:7357/assets/test-loader.js:60:16
> loadModules@http://localhost:7357/assets/test-loader.js:51:25
> load@http://localhost:7357/assets/test-loader.js:82:35
> http://localhost:7357/assets/test-support.js:6037:20: undefined
> is not an object (evaluating 'this.get')
> Log: |
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4290) Dockerfile improvements
[
https://issues.apache.org/jira/browse/TEZ-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283770#comment-17283770
]
Jonathan Turner Eagles commented on TEZ-4290:
-
+1 On these changes.
> Dockerfile improvements
> ---
>
> Key: TEZ-4290
> URL: https://issues.apache.org/jira/browse/TEZ-4290
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.10.1, 0.9.3
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Given these improvement ideas:
> https://github.com/aw-was-here/tez/commit/bead3a21e68ecf4a75d64d8c08b9f8fa0f98cf51
> thanks [~aw]
> Dockerfile changes could be easily tested, becuase it can be build locally
> (as discovered on TEZ-4283):
> {code}
> docker build -f ./build-tools/docker/Dockerfile ./build-tools/docker/
> {code}
> Moreover, it's applied on PR level, so a full precommit test (including all
> modules) could let us validate the correctness of the image changes.
> So, I'm suspecting that Dockerfile changes can come in advance, before
> introducing Jenkinsfile, which is a bit harder to test (I guess).
> according to [~aw]'s comments:
> {code}
> - Update to an image that won't be unsupported in a year
> - Remove a bunch of stuff that Tez doesn't use
> - Add support for a bunch of stuff that Yetus needs to test everything
> (however, keeping FindBugs over SpotBugs)
> - Make it Docker BuildKit compatible to make it build in parallel
> - Support both OpenJDK 8 and OpenJDK 11
> - Switch to python3 because python2 is effectively dead in January with pip
> changes
> - Remove the interactive bits of the Dockerfile since nothing appears to be
> using them
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4241) UI: containerLogs testWrapper test fails
[
https://issues.apache.org/jira/browse/TEZ-4241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283748#comment-17283748
]
Jonathan Turner Eagles commented on TEZ-4241:
-
[~abstractdog], [~euigeun_chung], I propose we put patch 02 in as a temporary
fix (under a different jira) until this test can be fixed properly so that
builds can get healthy again. Anyone against this idea?
> UI: containerLogs testWrapper test fails
>
>
> Key: TEZ-4241
> URL: https://issues.apache.org/jira/browse/TEZ-4241
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.10.1
>
> Attachments: TEZ-4241.01.patch, TEZ-4241.02.patch, TEZ-4241.03.patch,
> patch-unit-root.txt
>
>
> This causes failures in precommit for patches that are independent of tez ui.
> {code}
> not ok 750 PhantomJS 2.1 - Unit | Serializer | dag: containerLogs test
> ---
> actual: >
> null
> message: >
> Died on test #2
> testWrapper@http://localhost:7357/assets/test-support.js:6240:16
> test@http://localhost:7357/assets/test-support.js:6254:44
> http://localhost:7357/assets/tests.js:19223:24
> http://localhost:7357/assets/vendor.js:477:34
> tryFinally@http://localhost:7357/assets/vendor.js:358:21
> requireModule@http://localhost:7357/assets/vendor.js:475:15
> require@http://localhost:7357/assets/test-loader.js:60:16
> loadModules@http://localhost:7357/assets/test-loader.js:51:25
> load@http://localhost:7357/assets/test-loader.js:82:35
> http://localhost:7357/assets/test-support.js:6037:20: undefined
> is not an object (evaluating 'this.get')
> Log: |
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4290) Dockerfile improvements
[
https://issues.apache.org/jira/browse/TEZ-4290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17283741#comment-17283741
]
Jonathan Turner Eagles commented on TEZ-4290:
-
[~abstractdog], Is this jira ready to go in? Or is there more work still to do?
> Dockerfile improvements
> ---
>
> Key: TEZ-4290
> URL: https://issues.apache.org/jira/browse/TEZ-4290
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.10.1, 0.9.3
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Given these improvement ideas:
> https://github.com/aw-was-here/tez/commit/bead3a21e68ecf4a75d64d8c08b9f8fa0f98cf51
> thanks [~aw]
> Dockerfile changes could be easily tested, becuase it can be build locally
> (as discovered on TEZ-4283):
> {code}
> docker build -f ./build-tools/docker/Dockerfile ./build-tools/docker/
> {code}
> Moreover, it's applied on PR level, so a full precommit test (including all
> modules) could let us validate the correctness of the image changes.
> So, I'm suspecting that Dockerfile changes can come in advance, before
> introducing Jenkinsfile, which is a bit harder to test (I guess).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TEZ-4286) Consider sending fewer emails from github to d...@tez.apache.org
[
https://issues.apache.org/jira/browse/TEZ-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282696#comment-17282696
]
Jonathan Turner Eagles commented on TEZ-4286:
-
[~abstractdog], is git...@hive.apache.org a real mailing list? Also, if our
goal is to mimic the JIRA/gitbox emails. Would issues and pullrequests point to
issues@tez.apache.org to send PR comments and creation/closure of new PR? Agree
that d...@tez.apache.org is not the correct place for those.
I'm trying to imagine what equivalence would be, but would be open to
reimagining a new system that would better serve the community.
> Consider sending fewer emails from github to d...@tez.apache.org
> ---
>
> Key: TEZ-4286
> URL: https://issues.apache.org/jira/browse/TEZ-4286
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TEZ-4278 introduced .asf.yaml, where I haven't changed the defaults regarding
> notifications:
> https://gitbox.apache.org/schemes.cgi?tez
> {code}
> commits: comm...@tez.apache.org
> issues: d...@tez.apache.org
> pullrequests: d...@tez.apache.org
> {code}
> now looking at my emails and settings in hive project, I think
> d...@apache.tez.org should not be the recipient of so many mails from github,
> because it will kill the original intention of a dev list (I mean, getting
> relevant, useful emails from humans :) )
> for reference in hive there is:
> {code}
> commits: comm...@hive.apache.org
> issues: git...@hive.apache.org
> pullrequests: git...@hive.apache.org
> {code}
> I think the *pullrequests* part which really spams dev@ at the moment
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (TEZ-3966) Tez UI config couldn't be executed in browser
[ https://issues.apache.org/jira/browse/TEZ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-3966. - Fix Version/s: 0.9.3 0.10.1 Resolution: Fixed > Tez UI config couldn't be executed in browser > -- > > Key: TEZ-3966 > URL: https://issues.apache.org/jira/browse/TEZ-3966 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Roman Lukin >Priority: Major > Fix For: 0.10.1, 0.9.3 > > Attachments: Selection_043.png, Selection_044.png, > simple-cors-http-server.py > > Time Spent: 1h > Remaining Estimate: 0h > > Current name convention applied to the ui configuration file, produce the > following error in web browser - mime type check fails (which based on file > extesion) and if header `X-Content-Type-Options: nosniff` enabled config > wouldn't be executed and used. As a result we may have problems, for instance > links based on variable `yarnProtocol`. > More about header - > [https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options] > Possible solution - rename file `config.env` to `config.js` > [https://github.com/apache/tez/pull/26] > [https://github.com/rlukin/tez/commit/8db8c04faee04dc161c90c3961233ec5aa40518e] > > Steps to reproduce: > * Enable aforementioned header on web server > * Try to load tez ui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-3966) Tez UI config couldn't be executed in browser
[ https://issues.apache.org/jira/browse/TEZ-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282665#comment-17282665 ] Jonathan Turner Eagles commented on TEZ-3966: - +1. Disussion in github. Committing and cherry-picking to branch-0.9 > Tez UI config couldn't be executed in browser > -- > > Key: TEZ-3966 > URL: https://issues.apache.org/jira/browse/TEZ-3966 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Roman Lukin >Priority: Major > Attachments: Selection_043.png, Selection_044.png, > simple-cors-http-server.py > > Time Spent: 50m > Remaining Estimate: 0h > > Current name convention applied to the ui configuration file, produce the > following error in web browser - mime type check fails (which based on file > extesion) and if header `X-Content-Type-Options: nosniff` enabled config > wouldn't be executed and used. As a result we may have problems, for instance > links based on variable `yarnProtocol`. > More about header - > [https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options] > Possible solution - rename file `config.env` to `config.js` > [https://github.com/apache/tez/pull/26] > [https://github.com/rlukin/tez/commit/8db8c04faee04dc161c90c3961233ec5aa40518e] > > Steps to reproduce: > * Enable aforementioned header on web server > * Try to load tez ui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4278) Show PR link in Jira
[ https://issues.apache.org/jira/browse/TEZ-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4278. - Fix Version/s: 0.10.1 Resolution: Fixed +1. Committed to master via github. [~abstractdog], does this need to be cherry-picked to branch-0.9? I didn't think so based on the comments above. Let me know if needed. > Show PR link in Jira > > > Key: TEZ-4278 > URL: https://issues.apache.org/jira/browse/TEZ-4278 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 0.10.1 > > Attachments: Screen Shot 2021-01-29 at 6.52.59 PM.png > > Time Spent: 40m > Remaining Estimate: 0h > > In hive project, when a PR is created, link is automatically shown in Jira. I > got used to it, so it's very important :) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4130) Config for hard limiting the number of splits
[ https://issues.apache.org/jira/browse/TEZ-4130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275353#comment-17275353 ] Jonathan Turner Eagles commented on TEZ-4130: - [~abstractdog], now with TEZ-4271. Are we needing to increase the priority of this jira? Seems like unit test was needed, but perhaps some logic also. > Config for hard limiting the number of splits > - > > Key: TEZ-4130 > URL: https://issues.apache.org/jira/browse/TEZ-4130 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: TEZ-4130.01.patch, TEZ-4130.02.patch > > > During the investigation of a customer issue, I found that tez generated a > dag plan containing >4k tasks. It failed for hive because of bucket number > limitations (4k). It can be configured properly, e.g. bigger splits > (tez.grouping.min-size), but maybe it would be more convenient for users to > config a hard limit for the number of splits. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4266) Make Tez Inherit From Apache POM
[ https://issues.apache.org/jira/browse/TEZ-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4266. - Fix Version/s: 0.9.3 0.10.1 Resolution: Fixed merged to master from github and cherry-picked to branch-0.9. Thanks, [~belugabehr] for this patch. > Make Tez Inherit From Apache POM > > > Key: TEZ-4266 > URL: https://issues.apache.org/jira/browse/TEZ-4266 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 0.10.1, 0.9.3 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > This allows for inheriting all of the Apache Maven Plugins and makes plugins > easier to manage since all the versions are tested together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4278) Show PR link in Jira
[ https://issues.apache.org/jira/browse/TEZ-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275296#comment-17275296 ] Jonathan Turner Eagles commented on TEZ-4278: - This is some feedback and patch I received after the first round of github integration from yetus maintainer. I haven't had time to analyze the changes and try them out yet. https://github.com/aw-was-here/tez/commit/bead3a21e68ecf4a75d64d8c08b9f8fa0f98cf51 > Show PR link in Jira > > > Key: TEZ-4278 > URL: https://issues.apache.org/jira/browse/TEZ-4278 > Project: Apache Tez > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: Screen Shot 2021-01-29 at 6.52.59 PM.png > > > In hive project, when a PR is created, link is automatically shown in Jira. I > got used to it, so it's very important :) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4275) Use Google Guava Intern Facility
[ https://issues.apache.org/jira/browse/TEZ-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274157#comment-17274157 ] Jonathan Turner Eagles commented on TEZ-4275: - https://issues.apache.org/jira/browse/TEZ-2164 has details on shading guava and discussions > Use Google Guava Intern Facility > > > Key: TEZ-4275 > URL: https://issues.apache.org/jira/browse/TEZ-4275 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Google Guava has a pretty cool Interner facility. > > * More memory efficient than current offering. Map contains a weak key and > a static dummy value (current implementation uses a weak value) > * Current implementation has a single lock around the entire data structure. > Google segments their data structure into (default: 4) segments for better > concurrency > * All the other thoughtful stuff Google has added into this feature -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4275) Use Google Guava Intern Facility
[ https://issues.apache.org/jira/browse/TEZ-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274019#comment-17274019 ] Jonathan Turner Eagles commented on TEZ-4275: - Guava is a real headache to support as interoperability between version is a constant issue. Without classpath isolation from user jars or shading of guava library, adding new dependent libraries on guava I would be very hesitant about. > Use Google Guava Intern Facility > > > Key: TEZ-4275 > URL: https://issues.apache.org/jira/browse/TEZ-4275 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Google Guava has a pretty cool Interner facility. > > * More memory efficient than current offering. Map contains a weak key and > a static dummy value (current implementation uses a weak value) > * Current implementation has a single lock around the entire data structure. > Google segments their data structure into (default: 4) segments for better > concurrency > * All the other thoughtful stuff Google has added into this feature -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEZ-4266) Make Tez Inherit From Apache POM
[ https://issues.apache.org/jira/browse/TEZ-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274007#comment-17274007 ] Jonathan Turner Eagles commented on TEZ-4266: - +1. LGTM. I have verified class file to ensure version compatibility and that plugin version changes don't change behavior. I'll see if this can be easily cherry-picked to branch-0.9. Otherwise, I may ask for another branch-0.9 specific patch. > Make Tez Inherit From Apache POM > > > Key: TEZ-4266 > URL: https://issues.apache.org/jira/browse/TEZ-4266 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > This allows for inheriting all of the Apache Maven Plugins and makes plugins > easier to manage since all the versions are tested together. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TEZ-4269) Re-Work Threadpool in DAGAppMaster
[ https://issues.apache.org/jira/browse/TEZ-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Turner Eagles resolved TEZ-4269. - Fix Version/s: 0.10.1 Assignee: David Mollitor Resolution: Fixed Closing as merged through github to master. > Re-Work Threadpool in DAGAppMaster > -- > > Key: TEZ-4269 > URL: https://issues.apache.org/jira/browse/TEZ-4269 > Project: Apache Tez > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 0.10.1 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
