ambari git commit: AMBARI-20990 : writing query properly to hdfs file for handling unicode characters. (nitirajrathore)

2017-05-11 Thread nitiraj 
Repository: ambari
Updated Branches:
  refs/heads/branch-2.5 68704077b -> 57a6d15f6


AMBARI-20990 : writing query properly to hdfs file for handling unicode 
characters. (nitirajrathore)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/57a6d15f
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/57a6d15f
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/57a6d15f

Branch: refs/heads/branch-2.5
Commit: 57a6d15f6b1a140f062d15da75a6acaab1ccc77c
Parents: 6870407
Author: Nitiraj Singh Rathore 
Authored: Fri May 12 08:52:02 2017 +0530
Committer: Nitiraj Singh Rathore 
Committed: Fri May 12 08:52:02 2017 +0530

--
 .../ambari/view/hive2/resources/files/FileService.java   | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/57a6d15f/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
--
diff --git 
a/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
 
b/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
index 64880bb..88f8df1 100644
--- 
a/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
+++ 
b/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
@@ -185,9 +185,9 @@ public class FileService extends BaseService {
 try {
   filePath = sanitizeFilePath(filePath);
   LOG.debug("Rewriting file " + filePath);
-  FSDataOutputStream output = 
getSharedObjectsFactory().getHdfsApi().create(filePath, true);
-  output.writeBytes(request.file.getFileContent());
-  output.close();
+  HdfsApi hdfsApi = getSharedObjectsFactory().getHdfsApi();
+  HdfsUtil.putStringToFile(hdfsApi, filePath,
+  request.file.getFileContent());
   return Response.status(204).build();
 } catch (WebApplicationException ex) {
   throw ex;
@@ -209,7 +209,7 @@ public class FileService extends BaseService {
   try {
 FSDataOutputStream output = 
getSharedObjectsFactory().getHdfsApi().create(request.file.getFilePath(), 
false);
 if (request.file.getFileContent() != null) {
-  output.writeBytes(request.file.getFileContent());
+  output.write(request.file.getFileContent().getBytes());
 }
 output.close();
   } catch (FileAlreadyExistsException ex) {

ambari git commit: AMBARI-20990 : writing query properly to hdfs file for handling unicode characters. (nitirajrathore)

2017-05-11 Thread nitiraj 
Repository: ambari
Updated Branches:
  refs/heads/trunk 461209a90 -> fffd07c75


AMBARI-20990 : writing query properly to hdfs file for handling unicode 
characters. (nitirajrathore)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/fffd07c7
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/fffd07c7
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/fffd07c7

Branch: refs/heads/trunk
Commit: fffd07c75ba1af09b0ba3fec347bd7a6fff87807
Parents: 461209a
Author: Nitiraj Singh Rathore 
Authored: Fri May 12 08:52:02 2017 +0530
Committer: Nitiraj Singh Rathore 
Committed: Fri May 12 08:53:38 2017 +0530

--
 .../ambari/view/hive2/resources/files/FileService.java   | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/fffd07c7/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
--
diff --git 
a/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
 
b/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
index 64880bb..88f8df1 100644
--- 
a/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
+++ 
b/contrib/views/hive-next/src/main/java/org/apache/ambari/view/hive2/resources/files/FileService.java
@@ -185,9 +185,9 @@ public class FileService extends BaseService {
 try {
   filePath = sanitizeFilePath(filePath);
   LOG.debug("Rewriting file " + filePath);
-  FSDataOutputStream output = 
getSharedObjectsFactory().getHdfsApi().create(filePath, true);
-  output.writeBytes(request.file.getFileContent());
-  output.close();
+  HdfsApi hdfsApi = getSharedObjectsFactory().getHdfsApi();
+  HdfsUtil.putStringToFile(hdfsApi, filePath,
+  request.file.getFileContent());
   return Response.status(204).build();
 } catch (WebApplicationException ex) {
   throw ex;
@@ -209,7 +209,7 @@ public class FileService extends BaseService {
   try {
 FSDataOutputStream output = 
getSharedObjectsFactory().getHdfsApi().create(request.file.getFilePath(), 
false);
 if (request.file.getFileContent() != null) {
-  output.writeBytes(request.file.getFileContent());
+  output.write(request.file.getFileContent().getBytes());
 }
 output.close();
   } catch (FileAlreadyExistsException ex) {

ambari git commit: Hive view 2.0 : table stats parameters changed from Integer to Long to handle bigger values (Greg Senia via nitirajrathore)

2017-05-11 Thread nitiraj 
Repository: ambari
Updated Branches:
  refs/heads/trunk f7e9af87c -> 461209a90


Hive view 2.0 : table stats parameters changed from Integer to Long to handle 
bigger values (Greg Senia via nitirajrathore)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/461209a9
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/461209a9
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/461209a9

Branch: refs/heads/trunk
Commit: 461209a9050566117f7b7342be875d5cec7e94d8
Parents: f7e9af8
Author: Nitiraj Singh Rathore 
Authored: Fri May 12 08:44:49 2017 +0530
Committer: Nitiraj Singh Rathore 
Committed: Fri May 12 08:45:45 2017 +0530

--
 .../view/hive20/internal/dto/TableStats.java| 24 ++--
 .../internal/parsers/TableMetaParserImpl.java   |  8 +++
 2 files changed, 16 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/461209a9/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
--
diff --git 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
index 5d0f307..6af9359 100644
--- 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
+++ 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
@@ -34,11 +34,11 @@ public class TableStats {
 
   private DatabaseMetadataWrapper databaseMetadata;
   private Boolean isTableStatsEnabled;
-  private Integer numFiles;
-  private Integer numRows;
+  private Long numFiles;
+  private Long numRows;
   private String columnStatsAccurate;
-  private Integer rawDataSize;
-  private Integer totalSize;
+  private Long rawDataSize;
+  private Long totalSize;
 
   public Boolean getTableStatsEnabled() {
 return isTableStatsEnabled;
@@ -48,11 +48,11 @@ public class TableStats {
 isTableStatsEnabled = tableStatsEnabled;
   }
 
-  public Integer getNumFiles() {
+  public Long getNumFiles() {
 return numFiles;
   }
 
-  public void setNumFiles(Integer numFiles) {
+  public void setNumFiles(Long numFiles) {
 this.numFiles = numFiles;
   }
 
@@ -64,27 +64,27 @@ public class TableStats {
 this.columnStatsAccurate = columnStatsAccurate;
   }
 
-  public Integer getRawDataSize() {
+  public Long getRawDataSize() {
 return rawDataSize;
   }
 
-  public void setRawDataSize(Integer rawDataSize) {
+  public void setRawDataSize(Long rawDataSize) {
 this.rawDataSize = rawDataSize;
   }
 
-  public Integer getTotalSize() {
+  public Long getTotalSize() {
 return totalSize;
   }
 
-  public void setTotalSize(Integer totalSize) {
+  public void setTotalSize(Long totalSize) {
 this.totalSize = totalSize;
   }
 
-  public Integer getNumRows() {
+  public Long getNumRows() {
 return numRows;
   }
 
-  public void setNumRows(Integer numRows) {
+  public void setNumRows(Long numRows) {
 this.numRows = numRows;
   }
 

http://git-wip-us.apache.org/repos/asf/ambari/blob/461209a9/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
--
diff --git 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
index 711cab9..ef1b464 100644
--- 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
+++ 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
@@ -102,22 +102,22 @@ public class TableMetaParserImpl implements 
TableMetaParser {
 
 if(!Strings.isNullOrEmpty(numFiles) && 
!Strings.isNullOrEmpty(numFiles.trim())){
   tableStats.setTableStatsEnabled(true);
-  tableStats.setNumFiles(Integer.valueOf(numFiles.trim()));
+  tableStats.setNumFiles(Long.valueOf(numFiles.trim()));
 }
 
 if(!Strings.isNullOrEmpty(numRows) && 
!Strings.isNullOrEmpty(numRows.trim())){
   tableStats.setTableStatsEnabled(true);
-  tableStats.setNumRows(Integer.valueOf(numRows.trim()));
+  tableStats.setNumRows(Long.valueOf(numRows.trim()));
 }
 
 if(!Strings.isNullOrEmpty(rawDataSize) && 
!Strings.isNullOrEmpty(rawDataSize.trim())){
   tableStats.setTableStatsEnabled(true);
-  tableStats.setRawDataSize(Integer.valueOf(rawDataSize.trim()));
+  tableStats.setRawDataSize(Long.valueOf(rawDataSize.trim()));

ambari git commit: Hive view 2.0 : table stats parameters changed from Integer to Long to handle bigger values (Greg Senia via nitirajrathore)

2017-05-11 Thread nitiraj 
Repository: ambari
Updated Branches:
  refs/heads/branch-2.5 7f2f40e33 -> 68704077b


Hive view 2.0 : table stats parameters changed from Integer to Long to handle 
bigger values (Greg Senia via nitirajrathore)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/68704077
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/68704077
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/68704077

Branch: refs/heads/branch-2.5
Commit: 68704077bbcf7b4112b9b22c33dbd447d6aebc73
Parents: 7f2f40e
Author: Nitiraj Singh Rathore 
Authored: Fri May 12 08:44:49 2017 +0530
Committer: Nitiraj Singh Rathore 
Committed: Fri May 12 08:44:49 2017 +0530

--
 .../view/hive20/internal/dto/TableStats.java| 24 ++--
 .../internal/parsers/TableMetaParserImpl.java   |  8 +++
 2 files changed, 16 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/68704077/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
--
diff --git 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
index 5d0f307..6af9359 100644
--- 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
+++ 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/dto/TableStats.java
@@ -34,11 +34,11 @@ public class TableStats {
 
   private DatabaseMetadataWrapper databaseMetadata;
   private Boolean isTableStatsEnabled;
-  private Integer numFiles;
-  private Integer numRows;
+  private Long numFiles;
+  private Long numRows;
   private String columnStatsAccurate;
-  private Integer rawDataSize;
-  private Integer totalSize;
+  private Long rawDataSize;
+  private Long totalSize;
 
   public Boolean getTableStatsEnabled() {
 return isTableStatsEnabled;
@@ -48,11 +48,11 @@ public class TableStats {
 isTableStatsEnabled = tableStatsEnabled;
   }
 
-  public Integer getNumFiles() {
+  public Long getNumFiles() {
 return numFiles;
   }
 
-  public void setNumFiles(Integer numFiles) {
+  public void setNumFiles(Long numFiles) {
 this.numFiles = numFiles;
   }
 
@@ -64,27 +64,27 @@ public class TableStats {
 this.columnStatsAccurate = columnStatsAccurate;
   }
 
-  public Integer getRawDataSize() {
+  public Long getRawDataSize() {
 return rawDataSize;
   }
 
-  public void setRawDataSize(Integer rawDataSize) {
+  public void setRawDataSize(Long rawDataSize) {
 this.rawDataSize = rawDataSize;
   }
 
-  public Integer getTotalSize() {
+  public Long getTotalSize() {
 return totalSize;
   }
 
-  public void setTotalSize(Integer totalSize) {
+  public void setTotalSize(Long totalSize) {
 this.totalSize = totalSize;
   }
 
-  public Integer getNumRows() {
+  public Long getNumRows() {
 return numRows;
   }
 
-  public void setNumRows(Integer numRows) {
+  public void setNumRows(Long numRows) {
 this.numRows = numRows;
   }
 

http://git-wip-us.apache.org/repos/asf/ambari/blob/68704077/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
--
diff --git 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
index 711cab9..ef1b464 100644
--- 
a/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
+++ 
b/contrib/views/hive20/src/main/java/org/apache/ambari/view/hive20/internal/parsers/TableMetaParserImpl.java
@@ -102,22 +102,22 @@ public class TableMetaParserImpl implements 
TableMetaParser {
 
 if(!Strings.isNullOrEmpty(numFiles) && 
!Strings.isNullOrEmpty(numFiles.trim())){
   tableStats.setTableStatsEnabled(true);
-  tableStats.setNumFiles(Integer.valueOf(numFiles.trim()));
+  tableStats.setNumFiles(Long.valueOf(numFiles.trim()));
 }
 
 if(!Strings.isNullOrEmpty(numRows) && 
!Strings.isNullOrEmpty(numRows.trim())){
   tableStats.setTableStatsEnabled(true);
-  tableStats.setNumRows(Integer.valueOf(numRows.trim()));
+  tableStats.setNumRows(Long.valueOf(numRows.trim()));
 }
 
 if(!Strings.isNullOrEmpty(rawDataSize) && 
!Strings.isNullOrEmpty(rawDataSize.trim())){
   tableStats.setTableStatsEnabled(true);
-  tableStats.setRawDataSize(Integer.valueOf(rawDataSize.trim()));
+

ambari git commit: AMBARI-20999. More updates to llap configs for hdp stack (Siddharth via smohanty)

2017-05-11 Thread smohanty 
Repository: ambari
Updated Branches:
  refs/heads/trunk 023c819a3 -> f7e9af87c


AMBARI-20999. More updates to llap configs for hdp stack (Siddharth  via 
smohanty)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/f7e9af87
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/f7e9af87
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/f7e9af87

Branch: refs/heads/trunk
Commit: f7e9af87c75d8641a96b8b097b7b44f0a9f13538
Parents: 023c819
Author: Sumit Mohanty 
Authored: Thu May 11 17:03:27 2017 -0700
Committer: Sumit Mohanty 
Committed: Thu May 11 17:10:35 2017 -0700

--
 .../server/upgrade/UpgradeCatalog250.java   | 72 
 .../stacks/HDP/2.5/upgrades/config-upgrade.xml  | 15 
 .../HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml | 12 
 .../stacks/HDP/2.5/upgrades/upgrade-2.6.xml |  2 +
 .../configuration/hive-interactive-site.xml | 11 ++-
 .../HIVE/configuration/tez-interactive-site.xml | 13 
 .../server/upgrade/UpgradeCatalog250Test.java   | 23 +--
 7 files changed, 52 insertions(+), 96 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/f7e9af87/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
--
diff --git 
a/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
 
b/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
index 058acb2..1740e25 100644
--- 
a/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
+++ 
b/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
@@ -200,7 +200,6 @@ public class UpgradeCatalog250 extends 
AbstractUpgradeCatalog {
 updateHadoopEnvConfigs();
 updateKafkaConfigs();
 updateHIVEInteractiveConfigs();
-updateHiveLlapConfigs();
 unInstallAllZeppelinViews();
 updateZeppelinConfigs();
 updateAtlasConfigs();
@@ -495,56 +494,6 @@ public class UpgradeCatalog250 extends 
AbstractUpgradeCatalog {
 }
   }
 
-  protected void updateHiveLlapConfigs() throws AmbariException {
-AmbariManagementController ambariManagementController = 
injector.getInstance(AmbariManagementController.class);
-Clusters clusters = ambariManagementController.getClusters();
-
-if (clusters != null) {
-  Map clusterMap = clusters.getClusters();
-
-  if (clusterMap != null && !clusterMap.isEmpty()) {
-for (final Cluster cluster : clusterMap.values()) {
-  Set installedServices = cluster.getServices().keySet();
-
-  if (installedServices.contains("HIVE")) {
-Config hiveSite = 
cluster.getDesiredConfigByType(HIVE_INTERACTIVE_SITE);
-if (hiveSite != null) {
-  Map hiveSiteProperties = 
hiveSite.getProperties();
-  String schedulerDelay = 
hiveSiteProperties.get("hive.llap.task.scheduler.locality.delay");
-  if (schedulerDelay != null) {
-// Property exists. Change to new default if set to -1.
-if (schedulerDelay.length() != 0) {
-  try {
-int schedulerDelayInt = Integer.parseInt(schedulerDelay);
-if (schedulerDelayInt == -1) {
-  // Old default. Set to new default.
-  updateConfigurationProperties(HIVE_INTERACTIVE_SITE, 
Collections
-
.singletonMap("hive.llap.task.scheduler.locality.delay", "8000"), true,
-false);
-}
-  } catch (NumberFormatException e) {
-// Invalid existing value. Set to new default.
-updateConfigurationProperties(HIVE_INTERACTIVE_SITE, 
Collections
-  
.singletonMap("hive.llap.task.scheduler.locality.delay", "8000"), true,
-  false);
-  }
-}
-  }
-  updateConfigurationProperties(HIVE_INTERACTIVE_SITE,
-
Collections.singletonMap("hive.mapjoin.hybridgrace.hashtable", "true"), true,
-false);
-  updateConfigurationProperties("tez-interactive-site",
-
Collections.singletonMap("tez.session.am.dag.submit.timeout.secs", "1209600"), 
true,
-false);
-  // Explicitly skipping hive.llap.allow.permanent.fns during 
upgrades, since it's related to

ambari git commit: AMBARI-20999. More updates to llap configs for hdp stack (Siddharth via smohanty)

2017-05-11 Thread smohanty 
Repository: ambari
Updated Branches:
  refs/heads/branch-2.5 ef6c72da3 -> 7f2f40e33


AMBARI-20999. More updates to llap configs for hdp stack (Siddharth  via 
smohanty)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/7f2f40e3
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/7f2f40e3
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/7f2f40e3

Branch: refs/heads/branch-2.5
Commit: 7f2f40e33dd66e49192efd46c6e7a11c31b3c643
Parents: ef6c72d
Author: Sumit Mohanty 
Authored: Thu May 11 17:03:27 2017 -0700
Committer: Sumit Mohanty 
Committed: Thu May 11 17:03:27 2017 -0700

--
 .../server/upgrade/UpgradeCatalog250.java   | 73 
 .../stacks/HDP/2.5/upgrades/config-upgrade.xml  | 15 
 .../HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml | 12 
 .../stacks/HDP/2.5/upgrades/upgrade-2.6.xml |  2 +
 .../configuration/hive-interactive-site.xml | 11 ++-
 .../HIVE/configuration/tez-interactive-site.xml | 13 
 .../server/upgrade/UpgradeCatalog250Test.java   | 23 +-
 7 files changed, 52 insertions(+), 97 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/7f2f40e3/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
--
diff --git 
a/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
 
b/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
index c415a2b..53abe01 100644
--- 
a/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
+++ 
b/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
@@ -200,7 +200,6 @@ public class UpgradeCatalog250 extends 
AbstractUpgradeCatalog {
 updateHadoopEnvConfigs();
 updateKafkaConfigs();
 updateHIVEInteractiveConfigs();
-updateHiveLlapConfigs();
 unInstallAllZeppelinViews();
 updateZeppelinConfigs();
 updateAtlasConfigs();
@@ -762,57 +761,6 @@ public class UpgradeCatalog250 extends 
AbstractUpgradeCatalog {
 return content;
   }
 
-  protected void updateHiveLlapConfigs() throws AmbariException {
-AmbariManagementController ambariManagementController = 
injector.getInstance(AmbariManagementController.class);
-Clusters clusters = ambariManagementController.getClusters();
-
-if (clusters != null) {
-  Map clusterMap = clusters.getClusters();
-
-  if (clusterMap != null && !clusterMap.isEmpty()) {
-for (final Cluster cluster : clusterMap.values()) {
-  Set installedServices = cluster.getServices().keySet();
-
-  if (installedServices.contains("HIVE")) {
-Config hiveSite = 
cluster.getDesiredConfigByType(HIVE_INTERACTIVE_SITE);
-if (hiveSite != null) {
-  Map hiveSiteProperties = 
hiveSite.getProperties();
-  String schedulerDelay = 
hiveSiteProperties.get("hive.llap.task.scheduler.locality.delay");
-  if (schedulerDelay != null) {
-// Property exists. Change to new default if set to -1.
-if (schedulerDelay.length() != 0) {
-  try {
-int schedulerDelayInt = Integer.parseInt(schedulerDelay);
-if (schedulerDelayInt == -1) {
-  // Old default. Set to new default.
-  updateConfigurationProperties(HIVE_INTERACTIVE_SITE, 
Collections
-  
.singletonMap("hive.llap.task.scheduler.locality.delay", "8000"), true,
-  false);
-}
-  } catch (NumberFormatException e) {
-// Invalid existing value. Set to new default.
-updateConfigurationProperties(HIVE_INTERACTIVE_SITE, 
Collections
-
.singletonMap("hive.llap.task.scheduler.locality.delay", "8000"), true,
-false);
-  }
-}
-  }
-  updateConfigurationProperties(HIVE_INTERACTIVE_SITE,
-  
Collections.singletonMap("hive.mapjoin.hybridgrace.hashtable", "true"), true,
-  false);
-  updateConfigurationProperties("tez-interactive-site",
-  
Collections.singletonMap("tez.session.am.dag.submit.timeout.secs", "1209600"), 
true,
-  false);
-  // Explicitly skipping hive.llap.allow.permanent.fns during 
upgrades, since it's related to security,
-  // and we don't know if the value is set by the user or as a 
result of the previous default.
-}
-  }
-}
-  }
-}
-  }
-
-

ambari git commit: AMBARI-20977. Journalnode should support bulk restart o start or stop in hosts' page (zhangxiaolu via alejandro)

2017-05-11 Thread alejandro 
Repository: ambari
Updated Branches:
  refs/heads/trunk 1d46f23f7 -> 023c819a3


AMBARI-20977. Journalnode should support bulk restart o start or stop in hosts' 
page (zhangxiaolu via alejandro)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/023c819a
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/023c819a
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/023c819a

Branch: refs/heads/trunk
Commit: 023c819a3d05c5bd8fbf1933930fbc5ddfda6e16
Parents: 1d46f23
Author: Alejandro Fernandez 
Authored: Thu May 11 15:21:29 2017 -0700
Committer: Alejandro Fernandez 
Committed: Thu May 11 15:21:45 2017 -0700

--
 .../main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml   | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/023c819a/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
--
diff --git 
a/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml 
b/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
index 0606883..3e1a7ae 100644
--- 
a/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
+++ 
b/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
@@ -199,6 +199,10 @@
 PYTHON
 1200
   
+   
+  JournalNodes
+  NAMENODE
+  
   
 
   hdfs_journalnode

ambari git commit: AMBARI-20996 - Fallback Cluster Current Version Is Blank Which Causes Upgrade Errors (jonathanhurley)

2017-05-11 Thread jonathanhurley 
Repository: ambari
Updated Branches:
  refs/heads/branch-feature-AMBARI-12556 b8cb5d486 -> 7b0ccdae5


AMBARI-20996 - Fallback Cluster Current Version Is Blank Which Causes Upgrade 
Errors (jonathanhurley)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/7b0ccdae
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/7b0ccdae
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/7b0ccdae

Branch: refs/heads/branch-feature-AMBARI-12556
Commit: 7b0ccdae5572e4e96795f29f16dddb92c7e0128a
Parents: b8cb5d4
Author: Jonathan Hurley 
Authored: Thu May 11 13:46:14 2017 -0400
Committer: Jonathan Hurley 
Committed: Thu May 11 15:04:23 2017 -0400

--
 .../libraries/functions/stack_features.py   |  3 +-
 .../libraries/functions/version.py  | 31 +---
 .../upgrades/UpdateDesiredStackAction.java  |  8 +++--
 .../server/state/RepositoryVersionState.java|  4 +--
 ambari-server/src/test/python/TestVersion.py| 12 
 5 files changed, 41 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/7b0ccdae/ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
--
diff --git 
a/ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
 
b/ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
index 2b3df5f..cbd32e7 100644
--- 
a/ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
+++ 
b/ambari-common/src/main/python/resource_management/libraries/functions/stack_features.py
@@ -23,6 +23,7 @@ import ambari_simplejson as json
 from resource_management.core.exceptions import Fail
 from resource_management.core.logger import Logger
 from resource_management.libraries.functions.constants import Direction
+from resource_management.libraries.functions.version import 
format_stack_version
 
 # executionCommand for STOP
 _ROLE_COMMAND_STOP = 'STOP'
@@ -141,7 +142,7 @@ def get_stack_feature_version(config):
 if current_cluster_version is not None:
   version_for_stack_feature_checks = current_cluster_version
 elif original_stack is not None:
-  version_for_stack_feature_checks = original_stack
+  version_for_stack_feature_checks = format_stack_version(original_stack)
 else:
   version_for_stack_feature_checks = version if version is not None else 
stack_version
 

http://git-wip-us.apache.org/repos/asf/ambari/blob/7b0ccdae/ambari-common/src/main/python/resource_management/libraries/functions/version.py
--
diff --git 
a/ambari-common/src/main/python/resource_management/libraries/functions/version.py
 
b/ambari-common/src/main/python/resource_management/libraries/functions/version.py
index 2500430..406bd95 100644
--- 
a/ambari-common/src/main/python/resource_management/libraries/functions/version.py
+++ 
b/ambari-common/src/main/python/resource_management/libraries/functions/version.py
@@ -34,23 +34,32 @@ def _normalize(v, desired_segments=0):
   return [int(x) for x in v_list]
 
 
-def format_stack_version(input):
+def format_stack_version(value):
   """
-  :param input: Input string, e.g. "2.2" or "GlusterFS", or "2.0.6.GlusterFS", 
or "2.2.0.1-885"
+  :param value: Input string, e.g. "2.2" or "GlusterFS", or "2.0.6.GlusterFS", 
or "2.2.0.1-885"
   :return: Returns a well-formatted HDP stack version of the form #.#.#.# as a 
string.
   """
-  if input:
-if "-" in input:
-  input_array = input.split("-")
-  input = input_array[0]
+  if value:
+if "-" in value:
+  first_occurrence = value.find("-")
+  last_occurence = value.rfind("-")
 
-input = re.sub(r'^\D+', '', input)
-input = re.sub(r'\D+$', '', input)
-input = input.strip('.')
+  if first_occurrence == last_occurence:
+if value[0].isalpha():
+  value = value[first_occurrence + 1:]
+else:
+  value = value[:first_occurrence]
+  else:
+value = value[first_occurrence + 1:last_occurence]
 
-strip_dots = input.replace('.', '')
+
+value = re.sub(r'^\D+', '', value)
+value = re.sub(r'\D+$', '', value)
+value = value.strip('.')
+
+strip_dots = value.replace('.', '')
 if strip_dots.isdigit():
-  normalized = _normalize(str(input))
+  normalized = _normalize(str(value))
   if len(normalized) == 2:
 normalized = normalized + [0, 0]
   elif len(normalized) == 3:

http://git-wip-us.apache.org/repos/asf/ambari/blob/7b0ccdae/ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java

[ambari] Git Push Summary

2017-05-11 Thread jluniya 
Repository: ambari
Updated Branches:
  refs/heads/branch-feature-AMBARI-14714 [created] 1d46f23f7

ambari git commit: AMBARI-20992 - HDFS metainfo.xml has the wrong scope for component dependencies

2017-05-11 Thread tthorpe 
Repository: ambari
Updated Branches:
  refs/heads/trunk 7e2dd63b7 -> 1d46f23f7


AMBARI-20992 - HDFS metainfo.xml has the wrong scope for component dependencies


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/1d46f23f
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/1d46f23f
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/1d46f23f

Branch: refs/heads/trunk
Commit: 1d46f23f743bf64a3d086863af02d15aeb36f484
Parents: 7e2dd63
Author: Tim Thorpe 
Authored: Thu May 11 10:27:07 2017 -0700
Committer: Tim Thorpe 
Committed: Thu May 11 10:27:07 2017 -0700

--
 .../main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml | 6 +++---
 .../main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/1d46f23f/ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml
--
diff --git 
a/ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
b/ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml
index 3026938..0769ae8 100644
--- 
a/ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml
+++ 
b/ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml
@@ -35,7 +35,7 @@
   
 
   HDFS/ZKFC
-  host
+  cluster
   
 false
   
@@ -48,7 +48,7 @@
 
 
   ZOOKEEPER/ZOOKEEPER_SERVER
-  host
+  cluster
   
 false
   
@@ -61,7 +61,7 @@
 
 
   HDFS/JOURNALNODE
-  host
+  cluster
   
 false
   

http://git-wip-us.apache.org/repos/asf/ambari/blob/1d46f23f/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
--
diff --git 
a/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml 
b/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
index 967c974..0606883 100644
--- 
a/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
+++ 
b/ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/metainfo.xml
@@ -35,7 +35,7 @@
   
 
   HDFS/ZKFC
-  host
+  cluster
   
 false
   
@@ -48,7 +48,7 @@
 
 
   ZOOKEEPER/ZOOKEEPER_SERVER
-  host
+  cluster
   
 false
   
@@ -61,7 +61,7 @@
 
 
   HDFS/JOURNALNODE
-  host
+  cluster
   
 false

ambari git commit: AMBARI-20991 - Cluster Repository State Is Not Calculated Correctly For Service/Patch Repos (jonathanhurley)

2017-05-11 Thread jonathanhurley 
Repository: ambari
Updated Branches:
  refs/heads/branch-feature-AMBARI-12556 8938ea219 -> b8cb5d486


AMBARI-20991 - Cluster Repository State Is Not Calculated Correctly For 
Service/Patch Repos (jonathanhurley)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/b8cb5d48
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/b8cb5d48
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/b8cb5d48

Branch: refs/heads/branch-feature-AMBARI-12556
Commit: b8cb5d4863c73a1cd76a4b47fe792e650c2f67a2
Parents: 8938ea2
Author: Jonathan Hurley 
Authored: Thu May 11 11:30:31 2017 -0400
Committer: Jonathan Hurley 
Committed: Thu May 11 11:32:39 2017 -0400

--
 .../ClusterStackVersionResourceProvider.java| 38 +---
 1 file changed, 9 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/b8cb5d48/ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
--
diff --git 
a/ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
 
b/ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
index 1e49eb2..9ca8ddc 100644
--- 
a/ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
+++ 
b/ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
@@ -77,7 +77,6 @@ import 
org.apache.ambari.server.state.repository.VersionDefinitionXml;
 import org.apache.ambari.server.state.stack.upgrade.RepositoryVersionHelper;
 import org.apache.ambari.server.utils.StageUtils;
 import org.apache.ambari.server.utils.VersionUtils;
-import org.apache.commons.collections.CollectionUtils;
 import org.apache.commons.lang.StringUtils;
 import org.apache.commons.lang.math.NumberUtils;
 
@@ -240,6 +239,7 @@ public class ClusterStackVersionResourceProvider extends 
AbstractControllerResou
 
   RepositoryVersionEntity repositoryVersion = 
repositoryVersionDAO.findByPK(repositoryVersionId);
 
+  final List allStates = new ArrayList<>();
   final Map hostStates = new 
HashMap<>();
   for (RepositoryVersionState state: RepositoryVersionState.values()) {
 hostStates.put(state, new ArrayList());
@@ -248,10 +248,13 @@ public class ClusterStackVersionResourceProvider extends 
AbstractControllerResou
   StackEntity repoVersionStackEntity = repositoryVersion.getStack();
   StackId repoVersionStackId = new StackId(repoVersionStackEntity);
 
-  for (HostVersionEntity hostVersionEntity : 
hostVersionDAO.findByClusterStackAndVersion(
-  clusterName, repoVersionStackId, repositoryVersion.getVersion())) {
+  List hostVersionsForRepository = 
hostVersionDAO.findHostVersionByClusterAndRepository(
+  cluster.getClusterId(), repositoryVersion);
 
+  // create the in-memory structures
+  for (HostVersionEntity hostVersionEntity : hostVersionsForRepository) {
 
hostStates.get(hostVersionEntity.getState()).add(hostVersionEntity.getHostName());
+allStates.add(hostVersionEntity.getState());
   }
 
   setResourceProperty(resource, 
CLUSTER_STACK_VERSION_CLUSTER_NAME_PROPERTY_ID, clusterName, requestedIds);
@@ -259,35 +262,12 @@ public class ClusterStackVersionResourceProvider extends 
AbstractControllerResou
   setResourceProperty(resource, CLUSTER_STACK_VERSION_ID_PROPERTY_ID, 
repositoryVersion.getId(), requestedIds);
   setResourceProperty(resource, CLUSTER_STACK_VERSION_STACK_PROPERTY_ID, 
repoVersionStackId.getStackName(), requestedIds);
   setResourceProperty(resource, CLUSTER_STACK_VERSION_VERSION_PROPERTY_ID, 
repoVersionStackId.getStackVersion(), requestedIds);
-
+  setResourceProperty(resource, 
CLUSTER_STACK_VERSION_REPOSITORY_VERSION_PROPERTY_ID, 
repositoryVersion.getId(), requestedIds);
 
   @Experimental(feature = ExperimentalFeature.PATCH_UPGRADES,
   comment = "this is a fake status until the UI can handle services 
that are on their own")
-  RepositoryVersionState finalState = null;
-
-  for (RepositoryVersionState state : 
EnumSet.of(RepositoryVersionState.INSTALLING,
-  RepositoryVersionState.INSTALL_FAILED, 
RepositoryVersionState.OUT_OF_SYNC)) {
-
-if (CollectionUtils.isNotEmpty(hostStates.get(state))) {
-  finalState = state;
-  break;
-}
-  }
-
-  if (null == finalState) {
-int count = cluster.getClusterSize();
-
-for (RepositoryVersionState state :

[1/2] ambari git commit: AMBARI-20980. RU: RESTART SPARK2/SPARK2_CLIENT failed (dlysnichenko)

2017-05-11 Thread dmitriusan 
Repository: ambari
Updated Branches:
  refs/heads/branch-2.5 4aff24a91 -> ef6c72da3
  refs/heads/trunk 3edbc2c41 -> 7e2dd63b7


AMBARI-20980. RU: RESTART SPARK2/SPARK2_CLIENT failed (dlysnichenko)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/ef6c72da
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/ef6c72da
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/ef6c72da

Branch: refs/heads/branch-2.5
Commit: ef6c72da3ef1a9441ec8f374528ebad74a6e1f32
Parents: 4aff24a
Author: Lisnichenko Dmitro 
Authored: Thu May 11 17:29:10 2017 +0300
Committer: Lisnichenko Dmitro 
Committed: Thu May 11 17:29:10 2017 +0300

--
 .../common-services/SPARK/1.2.1/package/scripts/params.py  | 2 +-
 .../common-services/SPARK2/2.0.0/package/scripts/params.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/ef6c72da/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
--
diff --git 
a/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 
b/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
index cd556d7..fd473ad 100644
--- 
a/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
+++ 
b/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
@@ -212,7 +212,7 @@ dfs_type = default("/commandParams/dfs_type", "")
 # livy is only supported from HDP 2.5
 has_livyserver = False
 
-if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY, 
stack_version_formatted):
+if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY, 
stack_version_formatted) and "livy-env" in config['configurations']:
   livy_component_directory = 
Script.get_component_from_role(SERVER_ROLE_DIRECTORY_MAP, "LIVY_SERVER")
   livy_conf = format("{stack_root}/current/{livy_component_directory}/conf")
   livy_log_dir = config['configurations']['livy-env']['livy_log_dir']

http://git-wip-us.apache.org/repos/asf/ambari/blob/ef6c72da/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
--
diff --git 
a/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
 
b/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
index e508a0d..9f0c189 100755
--- 
a/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
+++ 
b/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
@@ -202,7 +202,7 @@ dfs_type = default("/commandParams/dfs_type", "")
 # livy for spark2 is only supported from HDP 2.6
 has_livyserver = False
 
-if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY2, 
stack_version_formatted):
+if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY2, 
stack_version_formatted) and "livy2-env" in config['configurations']:
   livy2_component_directory = 
Script.get_component_from_role(SERVER_ROLE_DIRECTORY_MAP, "LIVY2_SERVER")
   livy2_conf = format("{stack_root}/current/{livy2_component_directory}/conf")
   livy2_log_dir = config['configurations']['livy2-env']['livy2_log_dir']

[2/2] ambari git commit: AMBARI-20980. RU: RESTART SPARK2/SPARK2_CLIENT failed (dlysnichenko)

2017-05-11 Thread dmitriusan 
AMBARI-20980. RU: RESTART SPARK2/SPARK2_CLIENT failed (dlysnichenko)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/7e2dd63b
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/7e2dd63b
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/7e2dd63b

Branch: refs/heads/trunk
Commit: 7e2dd63b7a3b7509a28b12641de87122eda0e33e
Parents: 3edbc2c
Author: Lisnichenko Dmitro 
Authored: Thu May 11 17:31:58 2017 +0300
Committer: Lisnichenko Dmitro 
Committed: Thu May 11 17:31:58 2017 +0300

--
 .../common-services/SPARK/1.2.1/package/scripts/params.py  | 2 +-
 .../common-services/SPARK/2.2.0/package/scripts/params.py  | 2 +-
 .../common-services/SPARK2/2.0.0/package/scripts/params.py | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/7e2dd63b/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
--
diff --git 
a/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
 
b/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
index 6a59caf..42396bd 100644
--- 
a/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
+++ 
b/ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/params.py
@@ -211,7 +211,7 @@ dfs_type = default("/commandParams/dfs_type", "")
 # livy is only supported from HDP 2.5
 has_livyserver = False
 
-if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY, 
stack_version_formatted):
+if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY, 
stack_version_formatted) and "livy-env" in config['configurations']:
   livy_component_directory = 
Script.get_component_from_role(SERVER_ROLE_DIRECTORY_MAP, "LIVY_SERVER")
   livy_conf = format("{stack_root}/current/{livy_component_directory}/conf")
   livy_log_dir = config['configurations']['livy-env']['livy_log_dir']

http://git-wip-us.apache.org/repos/asf/ambari/blob/7e2dd63b/ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
--
diff --git 
a/ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
 
b/ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
index e60cab5..1d36a75 100644
--- 
a/ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
+++ 
b/ambari-server/src/main/resources/common-services/SPARK/2.2.0/package/scripts/params.py
@@ -200,7 +200,7 @@ dfs_type = default("/commandParams/dfs_type", "")
 # livy for spark2 is only supported from HDP 2.6
 has_livyserver = False
 
-if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY, 
stack_version_formatted):
+if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY, 
stack_version_formatted) and "livy-env" in config['configurations']:
   livy_component_directory = 
Script.get_component_from_role(SERVER_ROLE_DIRECTORY_MAP, "LIVY_SERVER")
   livy_conf = format("{stack_root}/current/{livy_component_directory}/conf")
   livy_log_dir = config['configurations']['livy-env']['livy_log_dir']

http://git-wip-us.apache.org/repos/asf/ambari/blob/7e2dd63b/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
--
diff --git 
a/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
 
b/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
index 4ed0718..1df3f2f 100755
--- 
a/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
+++ 
b/ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/params.py
@@ -198,7 +198,7 @@ dfs_type = default("/commandParams/dfs_type", "")
 # livy for spark2 is only supported from HDP 2.6
 has_livyserver = False
 
-if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY2, 
stack_version_formatted):
+if stack_version_formatted and check_stack_feature(StackFeature.SPARK_LIVY2, 
stack_version_formatted) and "livy2-env" in config['configurations']:
   livy2_component_directory = 
Script.get_component_from_role(SERVER_ROLE_DIRECTORY_MAP, "LIVY2_SERVER")
   livy2_conf = format("{stack_root}/current/{livy2_component_directory}/conf")
   livy2_log_dir = config['configurations']['livy2-env']['livy2_log_dir']

ambari git commit: AMBARI-20988. Add missing 'cluster_host_info' column to 'request' table to MSSQL DDL. (stoader)

2017-05-11 Thread stoader 
Repository: ambari
Updated Branches:
  refs/heads/branch-2.5 f125f6059 -> 4aff24a91


AMBARI-20988. Add missing 'cluster_host_info' column to 'request' table to 
MSSQL DDL. (stoader)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/4aff24a9
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/4aff24a9
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/4aff24a9

Branch: refs/heads/branch-2.5
Commit: 4aff24a915de9feed690769bbf8dab2bbb171d42
Parents: f125f60
Author: Toader, Sebastian 
Authored: Thu May 11 16:21:30 2017 +0200
Committer: Toader, Sebastian 
Committed: Thu May 11 16:21:30 2017 +0200

--
 ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/4aff24a9/ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql
--
diff --git a/ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 
b/ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql
index b78e2a4..e4127f8 100644
--- a/ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql
+++ b/ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql
@@ -351,6 +351,7 @@ CREATE TABLE request (
   request_schedule_id BIGINT,
   start_time BIGINT NOT NULL,
   status VARCHAR(255),
+  cluster_host_info VARBINARY(MAX) NOT NULL,
   CONSTRAINT PK_request PRIMARY KEY CLUSTERED (request_id),
   CONSTRAINT FK_request_schedule_id FOREIGN KEY (request_schedule_id) 
REFERENCES requestschedule (schedule_id));
 
@@ -362,7 +363,6 @@ CREATE TABLE stage (
   supports_auto_skip_failure SMALLINT DEFAULT 0 NOT NULL,
   log_info VARCHAR(255) NOT NULL,
   request_context VARCHAR(255),
-  cluster_host_info VARBINARY(MAX) NOT NULL,
   command_params VARBINARY(MAX),
   host_params VARBINARY(MAX),
   command_execution_type VARCHAR(32) NOT NULL DEFAULT 'STAGE',

ambari git commit: AMBARI-20872 Required properties for services in the blueprint are validated before the cluster provisioning is started (when the cluster creation template is posted)

2017-05-11 Thread lpuskas 
Repository: ambari
Updated Branches:
  refs/heads/trunk 55750b92a -> 3edbc2c41


AMBARI-20872 Required properties for services in the blueprint are validated 
before the cluster provisioning is started (when the cluster creation template 
is posted)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/3edbc2c4
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/3edbc2c4
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/3edbc2c4

Branch: refs/heads/trunk
Commit: 3edbc2c41d489c20d1f503723491461eaa328fab
Parents: 55750b9
Author: lpuskas 
Authored: Wed May 3 18:31:40 2017 +0200
Committer: lpuskas 
Committed: Thu May 11 13:35:40 2017 +0200

--
 .../server/topology/BlueprintValidatorImpl.java |  88 ++
 .../RequiredConfigPropertiesValidator.java  | 188 
 .../validators/TopologyValidatorFactory.java|   3 +-
 .../server/topology/BlueprintImplTest.java  |  13 -
 .../RequiredConfigPropertiesValidatorTest.java  | 302 +++
 5 files changed, 512 insertions(+), 82 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/3edbc2c4/ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
--
diff --git 
a/ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
 
b/ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
index 9688c60..2e5995c 100644
--- 
a/ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
+++ 
b/ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
@@ -20,7 +20,6 @@ package org.apache.ambari.server.topology;
 
 import java.util.ArrayList;
 import java.util.Collection;
-import java.util.Collections;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.Iterator;
@@ -49,6 +48,7 @@ public class BlueprintValidatorImpl implements 
BlueprintValidator {
 this.blueprint = blueprint;
 this.stack = blueprint.getStack();
   }
+
   @Override
   public void validateTopology() throws InvalidTopologyException {
 LOGGER.info("Validating topology for blueprint: [{}]", 
blueprint.getName());
@@ -57,7 +57,7 @@ public class BlueprintValidatorImpl implements 
BlueprintValidator {
 
 for (HostGroup group : hostGroups) {
   Map missingGroupDependencies = 
validateHostGroup(group);
-  if (! missingGroupDependencies.isEmpty()) {
+  if (!missingGroupDependencies.isEmpty()) {
 missingDependencies.put(group.getName(), missingGroupDependencies);
   }
 }
@@ -73,27 +73,24 @@ public class BlueprintValidatorImpl implements 
BlueprintValidator {
   cardinalityFailures.addAll(verifyComponentInAllHostGroups(component, 
autoDeploy));
 } else {
   cardinalityFailures.addAll(verifyComponentCardinalityCount(
-  component, cardinality, autoDeploy));
+component, cardinality, autoDeploy));
 }
   }
 }
 
-if (! missingDependencies.isEmpty() || ! cardinalityFailures.isEmpty()) {
+if (!missingDependencies.isEmpty() || !cardinalityFailures.isEmpty()) {
   generateInvalidTopologyException(missingDependencies, 
cardinalityFailures);
 }
   }
 
   @Override
   public void validateRequiredProperties() throws InvalidTopologyException {
-//todo: combine with RequiredPasswordValidator
-Map> missingProperties =
-  new HashMap<>();
 
 // we don't want to include default stack properties so we can't just use 
hostGroup full properties
 Map> clusterConfigurations = 
blueprint.getConfiguration().getProperties();
 
 // we need to have real passwords, not references
-if(clusterConfigurations != null) {
+if (clusterConfigurations != null) {
   StringBuilder errorMessage = new StringBuilder();
   boolean containsSecretReferences = false;
   for (Map.Entry> configEntry : 
clusterConfigurations.entrySet()) {
@@ -104,16 +101,16 @@ public class BlueprintValidatorImpl implements 
BlueprintValidator {
 String propertyValue = propertyEntry.getValue();
 if (propertyValue != null) {
   if (SecretReference.isSecret(propertyValue)) {
-errorMessage.append("  Config:").append(configType).append(" 
Property:").append(propertyName).append("\n");
+errorMessage.append("  Config:" + configType + " Property:" + 
propertyName + "\n");
 containsSecretReferences = true;
   }
 }
   }
 }
   }
-  if(containsSecretReferences) {
+

ambari git commit: AMBARI-20978 Ambari 3.0: Outstanding new components issues, additional patch. (atkach)

2017-05-11 Thread atkach 
Repository: ambari
Updated Branches:
  refs/heads/trunk 4b3cbef39 -> 55750b92a


AMBARI-20978 Ambari 3.0: Outstanding new components issues, additional patch. 
(atkach)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/55750b92
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/55750b92
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/55750b92

Branch: refs/heads/trunk
Commit: 55750b92a4f7ad00d2a63211f8b34364b47df289
Parents: 4b3cbef
Author: Andrii Tkach 
Authored: Thu May 11 13:49:12 2017 +0300
Committer: Andrii Tkach 
Committed: Thu May 11 13:49:12 2017 +0300

--
 ambari-web/app/styles/theme/bootstrap-ambari.css | 2 ++
 ambari-web/app/styles/wizard.less| 6 --
 2 files changed, 2 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/55750b92/ambari-web/app/styles/theme/bootstrap-ambari.css
--
diff --git a/ambari-web/app/styles/theme/bootstrap-ambari.css 
b/ambari-web/app/styles/theme/bootstrap-ambari.css
index 7f95dab..77bb086 100644
--- a/ambari-web/app/styles/theme/bootstrap-ambari.css
+++ b/ambari-web/app/styles/theme/bootstrap-ambari.css
@@ -389,6 +389,8 @@ h2.table-title {
 }
 .table input[type="checkbox"] + label {
   position: relative;
+  line-height: 1.3em;
+  font-size: initial;
   top: 4px;
   margin-bottom: 0;
 }

http://git-wip-us.apache.org/repos/asf/ambari/blob/55750b92/ambari-web/app/styles/wizard.less
--
diff --git a/ambari-web/app/styles/wizard.less 
b/ambari-web/app/styles/wizard.less
index 407d965..130896dd 100644
--- a/ambari-web/app/styles/wizard.less
+++ b/ambari-web/app/styles/wizard.less
@@ -199,12 +199,6 @@
   color: #00688B;
 }
   }
-  #step4 {
-.checkbox > div {
-  line-height: 1.3em;
-  font-size: initial;
-}
-  }
   #step6 {
 .pre-scrollable {
   max-height: 440px;

ambari git commit: AMBARI-20434. Enhance Kafka Service Check. Additional changes (WangJie via dlysnichenko)

2017-05-11 Thread dmitriusan 
Repository: ambari
Updated Branches:
  refs/heads/trunk 242d9e76f -> 4b3cbef39


AMBARI-20434. Enhance Kafka Service Check. Additional changes (WangJie via 
dlysnichenko)


Project: http://git-wip-us.apache.org/repos/asf/ambari/repo
Commit: http://git-wip-us.apache.org/repos/asf/ambari/commit/4b3cbef3
Tree: http://git-wip-us.apache.org/repos/asf/ambari/tree/4b3cbef3
Diff: http://git-wip-us.apache.org/repos/asf/ambari/diff/4b3cbef3

Branch: refs/heads/trunk
Commit: 4b3cbef39edb37b8b393dd5b19c968223a282281
Parents: 242d9e7
Author: Lisnichenko Dmitro 
Authored: Thu May 11 13:27:06 2017 +0300
Committer: Lisnichenko Dmitro 
Committed: Thu May 11 13:27:06 2017 +0300

--
 .../KAFKA/0.10.0.3.0/configuration/kafka-broker.xml  |  2 +-
 .../0.10.0.3.0/package/scripts/service_check.py  | 15 ++-
 2 files changed, 11 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/ambari/blob/4b3cbef3/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/configuration/kafka-broker.xml
--
diff --git 
a/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/configuration/kafka-broker.xml
 
b/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/configuration/kafka-broker.xml
index b62b986..46c14c0 100644
--- 
a/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/configuration/kafka-broker.xml
+++ 
b/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/configuration/kafka-broker.xml
@@ -527,7 +527,7 @@
   
   
 delete.topic.enable
-false
+true
 Enables delete topic. Delete topic through the admin tool 
will have no effect if this config is turned off
 
   

http://git-wip-us.apache.org/repos/asf/ambari/blob/4b3cbef3/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/service_check.py
--
diff --git 
a/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/service_check.py
 
b/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/service_check.py
index 0f3a417..cb66f98 100644
--- 
a/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/service_check.py
+++ 
b/ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/service_check.py
@@ -40,12 +40,17 @@ class ServiceCheck(Script):
 topic_exists_cmd = format("{kafka_home}/bin/kafka-topics.sh --zookeeper 
{kafka_config[zookeeper.connect]} --topic {topic} --list")
 topic_exists_cmd_p = subprocess.Popen(topic_exists_cmd.split(" "), 
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 topic_exists_cmd_out, topic_exists_cmd_err = 
topic_exists_cmd_p.communicate()
-# run create topic command only if the topic doesn't exists
-if topic not in topic_exists_cmd_out:
-  create_topic_cmd = format("{kafka_home}/bin/kafka-topics.sh --zookeeper 
{kafka_config[zookeeper.connect]} --create --topic {topic} --partitions 1 
--replication-factor 1")
+
+delete_topic_cmd = format("{kafka_home}/bin/kafka-topics.sh --zookeeper 
{kafka_config[zookeeper.connect]} --delete --topic {topic}")
+create_topic_cmd = format("{kafka_home}/bin/kafka-topics.sh --zookeeper 
{kafka_config[zookeeper.connect]} --create --topic {topic} --partitions 1 
--replication-factor 1")
+if topic in topic_exists_cmd_out:
+  # run delete topic and recreate the topic command only if the topic 
exists
+  command = source_cmd + " ; " + delete_topic_cmd + ";" + create_topic_cmd
+else:
+  # run create topic command 
   command = source_cmd + " ; " + create_topic_cmd
-  Logger.info("Running kafka create topic command: %s" % command)
-  call_and_match_output(command, 
format("({create_topic_cmd_created_output})|({create_topic_cmd_exists_output})"),
 "Failed to check that topic exists", user=params.kafka_user)
+Logger.info("Running kafka create topic command: %s" % command)
+call_and_match_output(command, 
format("({create_topic_cmd_created_output})|({create_topic_cmd_exists_output})"),
 "Failed to check that topic exists", user=params.kafka_user)
 
   def read_kafka_config(self):
 import params