[PR] RANGER-4750: Optimizations in the docker image build process [ranger]
kumaab opened a new pull request, #302:
URL: https://github.com/apache/ranger/pull/302
## What changes were proposed in this pull request?
Restructure the downloads of service tarballs from `download-archives.sh` to
`Dockerfile.ranger-service` as part of the image build for respective services.
Key highlights:
- When building specific images (eg: ranger-hadoop, ranger-hive), only
hadoop and hive tarballs are downloaded bringing down the time taken to bring
up the containers effectively.
- Allows to take advantage of docker parallelization thereby allowing
parallel service tarball downloads. Currently, the service tarball downloads
happen in a sequential manner with `download-archives.sh`.
- Although subject to network speeds, this also has shown minor improvements
in overall build times for all images.
## How was this patch tested?
Command to bring up containers:
- `docker-compose -f docker-compose.ranger-base.yml -f
docker-compose.ranger.yml -f docker-compose.ranger-${RANGER_DB_TYPE}.yml -f
docker-compose.ranger-usersync.yml -f docker-compose.ranger-tagsync.yml -f
docker-compose.ranger-kms.yml -f docker-compose.ranger-hadoop.yml -f
docker-compose.ranger-hbase.yml -f docker-compose.ranger-kafka.yml -f
docker-compose.ranger-hive.yml -f docker-compose.ranger-trino.yml -f
docker-compose.ranger-knox.yml build`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@ranger.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (RANGER-4750) [docker]: Optimize docker image build times
Abhishek Kumar created RANGER-4750: -- Summary: [docker]: Optimize docker image build times Key: RANGER-4750 URL: https://issues.apache.org/jira/browse/RANGER-4750 Project: Ranger Issue Type: Bug Components: Ranger Affects Versions: 2.4.0 Reporter: Abhishek Kumar Assignee: Abhishek Kumar Move service level downloads (*.tar.gz, *.jar) required by different services in their image build process from download-archives.sh to their own Dockerfiles. This also prevents unnecessary tar downloads for building specific images when running this step from README: chmod +x download-archives.sh && ./download-archives.sh Also, this prevents files from getting downloaded on the localhost in the downloads directory which reduces context sent to docker daemon during image build. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4749) Tag: Create an api to check if a resource is already available in database by resource input. And send tag list along with the resource list in response of GET /service/
[
https://issues.apache.org/jira/browse/RANGER-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Nadar updated RANGER-4749:
Description:
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource as well in
/service/tags/resources/paginated api.*
This will get the service resources with all its associated tags in a single
api which is also paginated.
And a free text search on the resource.
was:
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource in
/service/tags/resources/paginated api.*
This will get the service resources with all its associated tags in a single
api which is also paginated.
And a free text search on the resource.
> Tag: Create an api to check if a resource is already available in database by
> resource input. And send tag list along with the resource list in response of
> GET /service/tags/resources/paginated API
> -
>
> Key: RANGER-4749
> URL: https://issues.apache.org/jira/browse/RANGER-4749
> Project: Ranger
> Issue Type: Task
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> *1. Need an api to get the resource by resource if it exists in database.*
> Case:
> Suppose there already exists below tag and resource association.
> PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> Now we dont know about the above resource already existing, so we are now
> trying to create a new tag resource association as below
> PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> This is done using PUT /service/tags/importservicetags -> op: add_or_update
> So this will result in the deletion of PII_NAME association with the
> resource, and creation of PII_EMP_NAME tag associated with the resource.
> So if we have this GET servi
[jira] [Updated] (RANGER-4749) Tag: Create an api to check if a resource is already available in database by resource input. And send tag list along with the resource list in response of GET /service/
[
https://issues.apache.org/jira/browse/RANGER-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Nadar updated RANGER-4749:
Description:
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource as well in
/service/tags/resources/paginated api.*
Currently we are not returning the list of tags associated in the response.
This will get the service resources with all its associated tags in a single
api which is also paginated.
And a free text search on the resource.
was:
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource as well in
/service/tags/resources/paginated api.*
This will get the service resources with all its associated tags in a single
api which is also paginated.
And a free text search on the resource.
> Tag: Create an api to check if a resource is already available in database by
> resource input. And send tag list along with the resource list in response of
> GET /service/tags/resources/paginated API
> -
>
> Key: RANGER-4749
> URL: https://issues.apache.org/jira/browse/RANGER-4749
> Project: Ranger
> Issue Type: Task
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> *1. Need an api to get the resource by resource if it exists in database.*
> Case:
> Suppose there already exists below tag and resource association.
> PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> Now we dont know about the above resource already existing, so we are now
> trying to create a new tag resource association as below
> PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> This is done using PUT /service/tags/importservicetags -> op: add_or_update
> So this will result in the deletion of PII_NAME association with the
> resource, and creat
[jira] [Updated] (RANGER-4749) Tag: Create an api to check if a resource is already available in database by resource input. And send tag list along with the resource list in response of GET /service/
[
https://issues.apache.org/jira/browse/RANGER-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Nadar updated RANGER-4749:
Description:
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource in
/service/tags/resources/paginated api.*
This will get the service resources with all its associated tags in a single
api which is also paginated.
And a free text search on the resource.
was:
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource in
/service/tags/resources/paginated api.*
This will get the service resources with all its associated tags in a single
api which is also paginated.
> Tag: Create an api to check if a resource is already available in database by
> resource input. And send tag list along with the resource list in response of
> GET /service/tags/resources/paginated API
> -
>
> Key: RANGER-4749
> URL: https://issues.apache.org/jira/browse/RANGER-4749
> Project: Ranger
> Issue Type: Task
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> *1. Need an api to get the resource by resource if it exists in database.*
> Case:
> Suppose there already exists below tag and resource association.
> PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> Now we dont know about the above resource already existing, so we are now
> trying to create a new tag resource association as below
> PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> This is done using PUT /service/tags/importservicetags -> op: add_or_update
> So this will result in the deletion of PII_NAME association with the
> resource, and creation of PII_EMP_NAME tag associated with the resource.
> So if we have this GET service/tags/resource/service/\{serviceName}/resource
[jira] [Assigned] (RANGER-4749) Tag: Create an api to check if a resource is already available in database by resource input. And send tag list along with the resource list in response of GET /service
[
https://issues.apache.org/jira/browse/RANGER-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Nadar reassigned RANGER-4749:
---
Assignee: Anand Nadar
> Tag: Create an api to check if a resource is already available in database by
> resource input. And send tag list along with the resource list in response of
> GET /service/tags/resources/paginated API
> -
>
> Key: RANGER-4749
> URL: https://issues.apache.org/jira/browse/RANGER-4749
> Project: Ranger
> Issue Type: Task
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> *1. Need an api to get the resource by resource if it exists in database.*
> Case:
> Suppose there already exists below tag and resource association.
> PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> Now we dont know about the above resource already existing, so we are now
> trying to create a new tag resource association as below
> PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
> This is done using PUT /service/tags/importservicetags -> op: add_or_update
> So this will result in the deletion of PII_NAME association with the
> resource, and creation of PII_EMP_NAME tag associated with the resource.
> So if we have this GET service/tags/resource/service/\{serviceName}/resource
> api which will have the payload as
> {code:java}
> {
>"database": {
>"values": [
>"HR-DB"
>],
>"isRecursive": true
>},
>"table": {
>"values": [
>"EMPLOYEE"
>],
>"isRecursive": true
>},
>"column": {
>"values": [
>"NAME"
>],
>"isRecursive": true
>}
> } {code}
> we can check if this resource already exists or not.
> If the resource exists, we can get all the tags associated with that
> resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
> and send this in
> PUT /service/tags/importservicetags -> op: add_or_update
> then both PII_NAME and PII_EMP_NAME will be associated with the resource.
> *2. Need the list of associated tag names for a resource in
> /service/tags/resources/paginated api.*
> This will get the service resources with all its associated tags in a single
> api which is also paginated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4749) Tag: Create an api to check if a resource is already available in database by resource input. And send tag list along with the resource list in response of GET /service/
Anand Nadar created RANGER-4749:
---
Summary: Tag: Create an api to check if a resource is already
available in database by resource input. And send tag list along with the
resource list in response of GET /service/tags/resources/paginated API
Key: RANGER-4749
URL: https://issues.apache.org/jira/browse/RANGER-4749
Project: Ranger
Issue Type: Task
Components: admin
Reporter: Anand Nadar
*1. Need an api to get the resource by resource if it exists in database.*
Case:
Suppose there already exists below tag and resource association.
PII_NAME tag is associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
Now we dont know about the above resource already existing, so we are now
trying to create a new tag resource association as below
PII_EMP_NAME associated with (database:HR-DB, table:EMPLOYEE, column:NAME)
This is done using PUT /service/tags/importservicetags -> op: add_or_update
So this will result in the deletion of PII_NAME association with the resource,
and creation of PII_EMP_NAME tag associated with the resource.
So if we have this GET service/tags/resource/service/\{serviceName}/resource
api which will have the payload as
{code:java}
{
"database": {
"values": [
"HR-DB"
],
"isRecursive": true
},
"table": {
"values": [
"EMPLOYEE"
],
"isRecursive": true
},
"column": {
"values": [
"NAME"
],
"isRecursive": true
}
} {code}
we can check if this resource already exists or not.
If the resource exists, we can get all the tags associated with that
resource(in our case PII_NAME), and include the new tag PII_EMP_NAME as well
and send this in
PUT /service/tags/importservicetags -> op: add_or_update
then both PII_NAME and PII_EMP_NAME will be associated with the resource.
*2. Need the list of associated tag names for a resource in
/service/tags/resources/paginated api.*
This will get the service resources with all its associated tags in a single
api which is also paginated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
